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EXECUTIVE  SUMMARY 


OBJECTIVES 

The  Supervised  On-the-Job  Extension  Training  (SOJET)  program  is  a 
new  approach  to  delivering  duty  position  training  to  a  unit  in  a  non¬ 
resident  mode.  The  program's  delivery  system  contains  features  that 
differ  considerably  from  those  used  with  traditional  courses.  At  pre¬ 
sent  the  Army  useSr  the  program  to  administer  16  courses  designed  for 
operations  and  intelligence  enlisted  personnel  in  combat  battalions. 

A  field  test  of  the  SOJET  program  was  started  in  the  spring  of 
1978.  At  that  time  the  program's  delivery  system  had  not  been 
debugged,  and  portions  of  the  program's  administrative  procedures  were 
not  compatible  with  those  used  by  the  Institute  for  Professional 
Development  (I PD),  the  agency  responsible  for  administering  the  Army's 
Correspondence  Course  Program. 

The  purpose  of  the  present  research  was  threefold:  (a)  to  con¬ 
duct  a  formative  evaluation  of  the  prototype  SOJET  program;  (b)  to 
redesign  the  program's  centralized  administrative  procedures  to  make 
them  compatible  with  those  used  by  IPD;  and  (c)  to  complete  the  design 
of  the  SOJET  program  by  developing  a  feedback  system,  primarily  for 
course  developers.  In  addition,  the  research  project  intended  to  pro¬ 
vide  baseline  data  for  use  as  bench  marks  when  evaluating  future 
extension  raining  or  correspondence  programs. 


METHODOLOGY 

Interviews  with  and  surveys  of  actual  and  potential  students  and 
course  supervisors,  in  both  active  Army  and  Army  Reserve/National  Guard, 
were  used  to  assess  field  acceptance  of  the  program  and  to  identify 
program  features  in  need  of  change.  Training  records  were  analyzed  to 
identify  the  characteristics  of  persons  who  enrolled  in  the  program 
and  the  rate  at  which  they  progressed  through  their  program  of  study. 

The  administrative  feasibility  of  the  program  was  assessed 
through  discussions  with  personnel  from  the  Institute  for  Professional 
Development  (IPD).  These  discussions  provided  a  means  of  identifying 
features  of  the  SOJET  central  mangagement  procedures  which  administra¬ 
tors  considered  troublesome  and/or  time-consuming.  Solutions  to  these 
problems  were  developed  and  implemented  during  the  project  when 
possible. 

Interviews  with  course  developers  at  each  of  the  Army's  Combat 
Arms  schools  provided  information  which  was  used  to  design  a  feedback 


system  for  course  developers.  During  in-depth  interviews  with  IPD 
personnel  information  was  obtained  on  the  characteristics  of  a  new 
computer-based  management  system  soon  to  be  implemented  at  IPD.  SOJET 
procedures  then  were  revised  to  make  them  compatible  with  planned-for 
computer  support  capabilities. 

Program  costs  were  determined  by  analyzing  administrative  actions 
to  identify  the  labor  and  other  costs  associated  with  each  action. 


FINDINGS  AND  CONCLUSIONS 

(1)  The  SOJET  program  requires  that  a  course  supervisor  register 
along  with  a  student.  This  course  supervisor  helps  the  student  plan 
his  course  of  study,  administers  and  scores  tests,  reviews  test  fin¬ 
dings  with  the  student,  and  reports  test  results  to  IPD.  Both  actual 
and  potential  supervisors  and  students  reported  their  acceptance  of 
the  course  supervisor  concept.  Whether  the  requirement  to  have  a 
registered  course  supervisor  resulted  in  more  effective  training, 
faster  student  progress  through  SOJET  courses,  and/or  higher  course 
completion  rates  could  not  be  determined.  It  was  concluded  that  in 
the  absence  of  conclusive  data,  the  decision  to  continue  or  to  dis¬ 
continue  the  registered  supervisor  requirement  would  have  to  be  based 
on  policy. 

(2)  In  the  SOJET  program  subcou,se  tests  are  sent  directly  to  the 
course  supervisor.  Course  supervisors  reported  that  this  practice 
seldom  caused  security  or  storage  problems.  However,  it  does  increase 
mailing  and  labor  costs  by  a  small  amount.  It  was  concluded  that  the 
approach  is  an  acceptable  one  for  delivering  test  material  and  admi¬ 
nistrative  instruction?,  to  field  personnel.  Furthermore,  it  emphasi¬ 
zes  to  the  supervisor  that  he  is  responsible  for  the  student’s  training. 

(3)  The  SOJET  program  employs  a  number  of  procedures  designed  to 
minimize  the  time  required  of  supervisors  to  conduct  OJT.  For  each 
SOJET  course  a  Student  Training  Plan  was  provided.  These  plans  were 
designed  to  help  course  supervisors  and  students  identify  student  work 
and  training  requirements.  Comments  from  survey  respondents,  plus  an 
analysis  of  Student  Training  Plan  data,  indicated  that  the  plans  often 
were  not  being  used  as  intended--to  discriminate  between  the  level  of 
skill  required  on  the  job  and  present  skill  level  of  a  potential  or 
actual  job  encumbent.  Therefore,  it  was  suggested  that  the  use  of 
STPs  be  discontinued. 


Both  students  and  supervisors  reported  that  the  instructional 
material  was  of  good  quality;  course  supervisors  reported  that  parti¬ 
cipations  in  the  SOJET  program  reduced  the  time  required  of  them  to 
conduct  OJT. 


Procedures  for  returning  test  scores  to  IPD  so  that  training 
records  could  be  centrally  maintained  were  found  to  be  time  consuming 
and  cumbersome.  Revised,  simplified  test  reporting  procedures  were 
developed  and  implemented. 

(4)  Unlike  traditional  correspondence  courses,  all  SOJET  subcourse 
lesson  materials  requested  by  a  student  are  mailed  to  him  at  one  time. 
The  corresponding  subcourse  test  materials  are  sent  to  the  course 
supervisor  at  the  same  time.  This  approach  has  been  termed  "one-shot" 
mailing;  its  purpose  is  to  allow  students  and  supervisors  to  sequence 
their  program  of  study  in  accordance  with  unit  requirements. 

According  to  survey  data,  this  approach  was  acceptable  to 
enrolled  students  and  registered  supervisors.  However,  only  a  third 
of  the  students  and  supervisors  reported  taking  advantage  of  the 
sequencing  opportunities  afforded  by  this  approach.  An  analysis  of 
training  records  showed  that  many  students  studied  subcourse  material 
in  the  order  in  which  the  subcourses  were  numbered. 

A  cost  analysis  revealed  that  the  "one-shot"  mailing  is  somewhat 
more  expensive  than  the  approach  used  to  mail  out  material  for  tradi-  . 
tional  correspondence  courses.  Offsetting  this  finding  were  survey 
reports  that  SOJET  material  often  is  used  as  reference  aids  or  to  pre¬ 
pare  for  Skill  Qualification  Tests  by  persons  not  enrolled  in  the 
program.  It  was  suggested  that  this  last  finding  is  a  benefit  that 
may  more  than  offset  the  higher  mailing  costs  of  the  SOJET  approach. 

It  was  concluded,  therefore,  that  the  “one-shot"  mailing  should  con¬ 
tinue  to  be  used  with  SOJET  courses. 

(5)  The  testing  procedures  in  the  SOJET  program  are  considerably 
different  from  those  used  with  many  correspondence  courses.  Subcourse 
tests  are  performance-oriented,  and  are  admininstered  and  scored  by 
the  course  supervisor.  The  supervisor  uses  a  test  scoring  guide  that 
provides  detailed  instructions  on  how  to  score  each  test.  Also,  each 
subcourse  contains  a  pretest  that  if  passed,  allows  the  student  to 
receive  credit  for  the  subcourse  without  studying  the  lesson  material. 

Most  course  supervisors  commented  favorably  about  the  scoring 
guides,  although  they  noted  that  some  guides  contain  errors  in  need  of 
correction.  Many  supervisors  did  not  report  pretests  results  to  IPD 
unless  the  student  passed  all  requirements  on  the  pretest.  Also, 
supervisors  complained  about  the  forms  and  procedures  used  to  report 
test  results  to  IPD.  In  response  to  these  criticisms,  the  forms  and 
procedures  were  revised;  the  requirement  to  submit  pretest  results 
separate  from  posttest  scores  was  discontinued.  The  new  procedures 
are  easier  to  follow  and  use  the  same  test  reporting  form  used  for 
correspondence  courses  administered  by  IPD. 


vii 


(6)  Course  supervisors  are  responsible  for  giving  students 
immediate  feedback  about  test  results  and  for  providing  guidance 
regarding  what  material  the  student  should  restudy  if  he  or  she  failed 
portions  of  a  test*  Most  supervisors  and  students  reported  that  this 
practice  was  being  followed. 

(7)  The  original  SO JET  program  was  incomplete  in  that  procedures 
for  providing  feedback  to  course  developers  and  to  program  managers 
had  not  been  developed  at  the  time  the  program  was  implemented.  As 
part  of  this  effort  procedures  for  additional  feedback  reports  were 
prepared.  When  implemented  these  procedures  will  generate  reports 
dealing  with  (a)  student  profiles,  (b)  subcourse  cost  effectiveness, 
and  (c)  rate  of  subcourse  utilization  and  test  submission.  These  are 
in  addition  to  an  item  analysis  report  now  prepared  fay  IPO.  Such 
reports  will  provide  valuable  information  to  course  developers. 

(8)  Considerable  emphasis  was  placed  on  identifying  the  probable 
impact  of  personnel  turbulence  upon  student  progress  in  SOJET  program 
courses.  Interviews  with  field  personnel  suggested  that  the  turnover 
rate  within  battalion  S2  and  S3  sections  is  exceptionally  high.  A 
telephone  survey  of  active  Amy  units  within  COHUS  indicated  that  most 
SOJET  program  students  and  supervisors  can  expect  to  be  reassigned 
after  six  to  nine  months  on  the  job.  Analysis  of  training  records 
revealed  that  active  Amy  students  had  a  lower  program  completion  rate 
than  did  students  from  National  Guard  (KG)  or  U.  S.  Amy  Reserve 
(USAR)  units.  However,  in  terms  of  percentage  of  program  completed, 
active  Amy  students  completed  a  much  higher  percentage  of  their 
enrol led-for  program  than  did  NG/USAK  students. 

On  the  basis  of  these  findings,  it  was  concluded  that  personnel  tur¬ 
bulence  did  have  a  negative  impact  on  the  progress  of  active  Amy  stu- 
de-.:s  and  on  course  completion  rates.  This  impact  probably  was  in¬ 
tensified  by  the  nature  of  SOJET  course  material,  trfiich  is  duty 
position-oriented.  When  a  person  is  assigned  to  a  new  duty  position 
there  is  little  Incentive  to  continue  In  the  SOJET  program.  On  the 
other  hand,  it  was  noted  that  the  task-oriented,  duty  position- 
oriented  nature  of  SOJET  lesson  material  Increases  the  likelihood  that 
students  will  benefit  from  study  of  selected  subcourscs  even  though  they 
may  never  complete  an  entire  SOJET  course. 

(9)  A  coat  comparison  of  the  SOJET  program  with  the  procedure?  weed 
to  centrally  administer  correspondence  courses  revealed  that  the  SCUET 
program  coats  327,  more  to  adminiatcr.  A  considerable  portion  of  the 
increased  coat  is  due  to  the  need  to  noil  teats  and  teat  scoring  guides 
for  each  SOJET  aubcourae  to  a  course  auperviaor.  It  was  concluded  that 
the  bulk  of  these  higher  coats  would  be  incurred  by  any  frogram  that 
depended  on  field  personnel  to  administer  and  score  tests. 
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(10)  It  was  estimated  that  the  extra  cost  of  using  SOJET  procedures 
(as  opposed  to  traditional  procedures)  to  manage  the  16  SOJET  program 
courses  for  one  year  would  be  $6,288,  assuming  an  annual  enrollment  of 
600  students.  In  view  of  this  estimate,  plus  the  finding  that  there 
is  a  perceived  need  for  SOJET  instructional  material,  it  was  concluded 
that  ACCP  should  continue  to  offer  all  SOJET  courses,  and  that  the 
courses  should  be  administered  by  SOJET  procedures  (revised)  at  least 
through  calendar  year  1980.  Also,  it  was  concluded  that  cost  con¬ 
siderations  should  not  deter  application  of  the  SOJET  program  to  a 
select  group  of  courses  where  the  administration  and  scoring  of  per¬ 
formance  tests  by  a  course  supervisor  is  appropriate.  It  was  noted 
also  that  the  task-oriented,  self-contained  nature  of  SOJET  lesson 
material  makes  that  material  especially  applicable  for  high  personnel 
turbulence  duty  po$itions--a  student  can  benefit  from  study  of  one  or 
a  few  subcourses  even  though  he  may  not  have  an  opportunity  to 
complete  an  entire  course. 
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EVALUATION  AND  DESIGN  OF  A  PROTOTYPE  SYSTEM  FOR  MANAGEMENT  OF 
SUPERVISED  ON-THE-JOB  EXTENSION  TRAINING  PROGRAM  (SOJET) 


I.  OBJECTIVES 


During  the  spring  of  1978  the  U.S.  Arrr\y  began  an  extensive  field 
test  of  a  new  way  of  delivering  duty-position  training  to  a  unit  in  a 
non-resident  mode.  This  program  was  known  as  the  Supervised  On-the- 
Job  Extension  Training  (SOJET)  program.  The  delivery  system  for  the 
program  contained  features  that  differed  considerably  from  those  used 
with  traditional  correspondence  courses.*  There  was  a  need,  there¬ 
fore  to  evaluate  the  cost  effectiveness  and  the  acceptability  of  those 
features  and  to  revise  them  as  appropriate. 

When  the  SOJET  program  was  initially  implemented,  certain  por¬ 
tions  had  to  be  administered  on  a  "management  by  exception"  basis, 
using  specially  developed  procedures.  There  was  a  need  to  redesign 
these  procedures  to  make  them  compatible  with  the  new  information 
management  system  to  be  adopted  by  the  Institute  for  Professional 
Development  (IPD),  the  agency  responsible  for  administering  the  Army 
Correspondence  Course  Program.  In  addition,  a  system  for  providing 
feedback  to  course  developers  and  managers  had  not  been  designed. 

The  present  study  was  undertaken  to  evaluate  the  delivery  system 
for  the  SOJET  program  and  to  complete  the  design  of  that  system.  The 
technical  objectives  of  the  study  were  to:  (a)  conduct  a  formative  eval¬ 
uation  of  the  prototype  extension  training  program,  and  (b)  complete  the 
design  of  an  extension  training  management  system  for  the  centralized 
administration  and  delivery  of  performance-oriented  extension  training. 


^Throughout  this  report  "traditional  correspondence  course"  refers  to 
courses  which  are  MOS  rather  than  duty-position  oriented,  and  do  not 
require  the  support  of  a  registered  course  supervisor.  Also,  most 
"traditional"  correspondence  courses  employ  paper  and  pencil  know¬ 
ledge  tests  which  are  centrally  scored,  require  students  to  study 
lesson  material  in  a  set  sequence,  and  send  to  the  student  only  a  few 
subcourse  lesson  packets  at  any  one  time. 
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II.  THE  SOJET  PROGRAM 


A.  PROGRAM  OBJECTIVES  AND  OPERATIONAL  APPROACH 

The  SOJET  training  delivery  system  is  based  on  five  system  objec¬ 
tives,  which  in  turn  are  based  on  Army  training  policy  and/or  training 
technology  practices  that  are  considered  to  promote  effective  training. 
These  five  system  objectives  are  listed  in  Figure  1,  along  with  the 
one  or  more  operational  approaches  taken  to  implement  each  objective. 

A  basic  tenet  of  the  SOJET  program  is  that  the  supervisor  must  be 
recognized  as  having  responsibility  for  the  training  of  those  under 
his/her  supervision.  All  Army  supervisors  have  this  responsibility 
but  in  many  instances  it  is  not  actively  exercised.  The  SOJET  program 
required  supervisors  to  take  an  active  role  in  the  training  of  their 
personnel.  For  example,  in  the  traditional  correspondence  course 
program,  the  student  is  responsible  for  enrolling  in  the  course  and 
pursuing  his  study.  In  the  SOJET  program,  the  course  supervisor  is 
responsible  for  helping  the  student  identify  his  training  requirements 
and  for  administering  and  scoring  tests.  To  emphasize  the  nature  of 
this  responsibility  the  supervisor  is  required  to  register  by  name 
along  with  the  student.  Also,  subcourse  tests  are  sent  directly  to 
the  course  supervisor. 

In  the  absence  of  duty-oriented  lesson  material  supervisory  per¬ 
sonnel  are  required  to  provide  on-the-job  training  as  best  they  can. 
Often  the  supervisor  must  devise  a  course  of  study,  actually  serve  as 
a  tutor,  and  maintain  student  training  records.  Those  supervisors  who 
conscientiously  fulfill  their  training  responsibilities  often  spend 
considerable  time  in  conducting  on-the-job  training.  The  SOJET 
program  was  designed  in  recognition  of  the  need  to  reduce  these  time 
demands.  To  accomplish  this  the  SOJET  program  provides  (a)  a  tool,  a 
student  training  plan,  which  can  be  used  to  identify  job  requirements 
and  student  training  requirements;  (b)  high  quality,  self-study  lesson 
material  keyed  to  tasks  actually  performed  on  the  job;  (c)  procedures 
for  maintaining  training  records  centrally. 

Within  units  some  mechanism  is  needed  whereby  persons  can  study 
or  ref ami  1  i arize  themselves  with  a  task  before  they  must  perform  it. 

In  the  SOJET  program,  this  unit  need  is  met  by  providing  a  "one-shot “ 
mailing  of  all  subcourses  requested  by  the  student.  The  supervisor 
and  the  student  are  then  free  to  study  the  course  material  in  the 
sequence  which  best  meets  their  unit  needs.  To  accomplish  this  the 
subcourses  had  to  be  constructed  so  that  they  were  self-contained 
training  packages  requiring  no  prerequisite  training. 
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Figure  1.  SOJET  Program  Management  System  Objectives  and  Operational  Approach 


Training  technology  research  has  demonstrated  conclusively  that 
learning  is  promoted  by  providing  immediate  feedback  to  the  student. 

In  the  traditional  correspondence  course  a  student  takes  an  examina¬ 
tion,  the  examination  is  centrally  scored,  and  some  four  to  six  weeks 
later  the  results  are  forwarded  to  the  student.  In  the  SOJET  program 
the  supervisor  administers  performance  tests  and  then  immediately  scores 
and  reviews  the  test  results  with  the  student.  The  supervisor  explains 
the  nature  of  any  test  errors  and  directs  the  student  to  restudy  those 
portions  of  the  lesson  material  dealing  with  the  requirements  he  did 
not  pass  on  the  test.  To  accomplish  this  system  objective,  the  sub¬ 
course  performance  test  is  designed  so  that  it  can  be  scored  by  super¬ 
visors.  In  addition,  a  standardized  scoring  guide  was  developed  for 
each  subcourse.  This  guide  identifies  correct  re-sponses,  prescribes 
minimum  performance  standards,  and  provides  directions  for  restudy. 

The  administration  of  a  correspondence  course  program  is  a  com¬ 
plicated  process  requiring  a  considerable  amount  of  feedback.  The 
Institute  for  Professional  Development  (IPD)  prepares  a  number  of 
management  reports  of  interest  to  program  managers.  However,  with  the 
exception  of  an  item  analysis  report,  IPD  does  not  produce  reports 
that  are  of  interest  to  course  developers.  As  shown  in  Figure  1,  some 
of  the  reports  that  would  be  useful  to  both  course  developers  and 
program  managers  would  deal  with  such  topics  as  (a)  the  relevance  of 
training  to  local  job  requirements,  (b)  the  extent  to  which  training 
is  reaching  the  target  audience,  (c)  the  cost  effectiveness  of  indivi¬ 
dual  subcourses,  (d)  subcourse  quality  control,  and  (e)  the  impact  of 
personnel  turbulence  on  training  delivery. 


B.  PROGRAM  CONTENT  AND  ORGANIZATION 

The  sixteen  SOJET  courses  developed  to  date  are  directed  to 
operations  (S2)  and  intelligence  (S3)  enlisted  personnel  duty  posi¬ 
tions,  within  combat  arms  battalions.  These  positions  were  selected 
as  the  vehicle  for  the  program  because  many  persons  assigned  to  bat¬ 
talion  S2  or  S3  positions  have  not  had  the  opportunity  to  take  formal 
instructions  in  operations  or  intelligence-related  tasks. 

The  typical  organization  of  a  SOJET  course  is  shown  in  Figure  2. 
Some  subcourses  cover  tasks  that  are  common  in  all  types  of  battalions. 
Also,  there  are  a  number  of  special  subcourses  unique  to  one  of  the 
four  combat  arms  branches.  As  a  further  illustration,  the  operations 
sergeant/assistant  operations  sergeant  lesson  material  is  packaged 
into  four  courses  (Y01-Y04),  a  separate  course  for  each  of  the  four 
combat  arms  branches(Table  1).  Fourteen  core  subcourses  are  common  to 
each  of  the  four  combat  arms  branches.  In  addition,  there  are  two  or 
more  special  subcourses  for  each  branch.  Thus,  a  person  enrolling  for 
the  armor  operations  sergeant  course  ( Y01 )  enrolls  for  a  total  of  19 
subcourses. 


Figure  2.  Organization  of  SOUET  Course  Material 


TABLE  1 


SUBCOURSES  AND  STUDENT  CREDIT  HOURS  FOR 
SOJET  OPERATIONS/INTELLIGENCE  COURSES 


COURSE 

AIR  DEFENSE 
ARTILERY 

ARMOR 

FIELD 

ARTILLERY 

INFANTRY 

OPERATIONS  SERGEANT/ 
ASSISTANT  OPERATIONS 
SERGEANT 

Course  Number  (Student 
Lesson  Packets) 

Y04 

Y01 

Y03 

Y02 

Number  of  Core  Sub¬ 
courses  (for  all 
Combat  Arms) 

14 

14 

14 

14 

Number  of  Special 
Subcourses  (specific 
to  a  Branch) 

5 

5 

4 

2 

Total  Number  of 
Subcourses 

19 

19 

18 

16 

Total  Student  Credit 
Hours 

53 

48 

49 

51 

INTELLIGENCE  SERGEANT/ 
ASSISTANT  INTELLIGENCE 
SERGEANT 

Course  Number  (Student 
Lesson  Packets) 

Y08 

Y05 

Y07 

Y06 

Number  of  Core  Sub¬ 
courses  (for  all 
Combat  Arms) 

11 

11 

11 

11 

Number  of  Specific 
Subcourses  (specific 
to  a  Branch) 

5 

4 

5 

3 

Total  Number  of  Sub¬ 
courses 

16 

IS 

16 

14 

Total  Student  Credit 
Hours 

39 

32 

35 

34 

OPERATIONS  ASSISTANT/ 
SPECIALIST 

Course  Number  (Student 
Lesson  Packets) 

Y12 

Y09 

YU 

Y10 

Number  of  Core  Sub¬ 
courses  (for  all 
Combat  Arms) 

4 

4 

4 

4 

Number  of  Special  Sub¬ 
courses  (specific 
to  a  Branch) 

6 

0 

0 

1 

Total  Number  of  Sub¬ 
courses 

10 

4 

4 

S 

Total  Student  Credit 
Hours 

44 

19 

19 

_ 25_ 

INTELLIGENCE  ASSISTANT/ 
SPECIALIST 

Course  Number  (Student 
Lesson  Packets) 

Y16 

Y13 

Y1S 

Y14 

Number  of  Core  Sub¬ 
courses  (for  all 
Corfcat  Arms) 

j 

7 

7 

7 

Number  of  Special 
Subcourses  (specific 
to  a  Branch) 

6 

0 

0 

1 

Total  Number  of  Sub- 
courses 

13 

7 

7 

8 

Total  Student  Credit 
Hours 

43 

23 

23 

25 

f 


As  shown  in  Figure  2,  subcourse  material  is  organized  into  two 
packets.  The  student  packet  contains  lesson  material  and  practical 
exercises.  The  supervisor's  test  packet  contains  pretest  material, 
posttest  material,  scoring  instructions,  and  instructions  and  forms 
for  recording  test  results  and  forwarding  them  to  the  Institute  of 
Professional  Development. 


C.  PROGRAM  IMPLEMENFAi lON 

The  SOJET  program  was  implemented  on  an  experimental  basis  during 
the  spring  of  1978.  To  hasten  the  process  of  implementation  and  to 
assure  that  suitable  numbers  of  students  would  be  available  during  the 
program  evaluation  phar-e  of  the  implementation,  an  extensive  publicity 
campaign  was  conducted  beginning  early  in  1978.  The  availability  of 
the  SOJET  program  was  announced  in  Combat  Arms  School  publications  and 
a  "GREEN  publicity  package"  was  sent  to  all  battalions  in  the  active 
Arn\y,  Army  Reserves,  and  National  Guard.  Beginning  in  May  1978, 
program  enrollment  applications  were  accepted. 

Persons  responding  to  this  publicity  were  sent  a  "Supervisor's 
Guide"  which  contained  information  about  the  program,  and  instructions 
and  forms  for  enrolling.  A  revised  version  of  this  document,  now 
called  a  "Course  Guide,"  is  contained  in  Annex  A. 
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III.  EVALUATION  AND  REVISION  METHODOLOGY 


The  evaluation,  revision  and  design  activities  engaged  in 
during  this  study  are  shown  in  Table  2.  These  activities  consisted 
of: 

o  Determination  of  field  acceptance  of  the  SOJET  program 

o  Evaluation  of  the  administrative  feasibility  of  the  SOJET 
program  delivery  system 

o  Evaluation  of  student  progress 

o  Determination  of  SOJET  program  costs 

o  Development  of  revisions  to  program  delivery  procedures 

o  Development  of  improved  feedback  procedures 

Each  of  these  activities  is  described  in  more  detail  on  the 
following  pages. 


A.  FIELD  ACCEPTANCE 

Information  on  field  acceptance  of  the  program  was  obtained  from  on¬ 
site  interviews  at  Arrny  posts  and  from  three  surveys  conducted  by  mail. 


Field  Visits.  From  November  1978  through  March  1979,  on-site 
interviews  were  held  at  five  installations--Forts  Campbell,  Hood, 

Riley  (two  visits),  Sill,  and  Stewart.  During  these  visits  three 
groups  of  persons  were  interviewed:  enrolled  students  and  registered 
supervisors,  potential  students  and  supervisors  who  had  requested 
information  about  the  program,  and  potential  students  and  supervisors 
who  were  unfamiliar  with  the  program. 

The  purpose  of  these  interviews  was  to  determine  receptivity  to 
the  program  by  potential  students  and  supervisors,  the  impact  of 
course  publicity  on  enrollment,  and  the  acceptance  of  various  features 
of  the  program's  delivery  system.  The  structured  interview  used 
during  these  visits  is  contained  in  Annex  A.  The  categories  and  num¬ 
bers  of  persons  interviewed  are  shown  in  Table  3,  and  the  interview 
data  are  summarized  in  Annex  A. 


Survey  of  Enrolled  Students  and  Registered  Supervisors  in 


National  Guard  and  Reserve  Units  (1st  Survey).  As  a  supplement  to 
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IMIS  *  Instructional  Management  Information  System 


aNumber  of  persons  interviewed 
^Number  of  survey  respondents 
cNational  Guard/U.S.  Army  Reserve 


the  field  visits,  questionnaires  were  developed  to  obtain  data  from 
students  and  course  supervisors  who  were  members  of  the  National  Guard 
or  U.S.  Army  Reserve.  These  questionnaires  (Annex  A)  were  sent  to  all 
NG/USAR  personnel  who  were  in  the  SOJET  program  as  of  1  January  1979 — 

137  enrolled  students  and  67  registered  supervisors.  The  rates  of 
return  from  enrolled  students  and  registered  supervisors  were  28%  and 
47%,  respectively.  The  data  obtained  are  presented  in  Annex  A. 

Survey  of  Persons  Who- Had  Requested  Information  on  the  Program 
but  Had  Not  Enrol led/Registered  (2nd  Survey).  Before  and  during  field 
tests  on  the  SOJET  OPS/INTEL  program,  the  program  was  publicized  in 
various  ways.  As  a  consequence  many  persons  requested  information 
about  the  program,  and  were  sent  a  Supervisor's  Guide  packet.  If 
these  persons  did  not  enroll  as  students  or  register  as  course  super¬ 
visors  within  two  months,  they  were  sent  a  questionnaire  (Annex  A). 

The  purpose  of  the  questionnaire  was  to  (a)  obtain  reader  reaction  to 
the  program  as  described  in  the  Program  Supervisor's  Guide,  and  (b) 
obtain  information  about  enrollment  intentions.  This  questionnaire 
was  sent  to  95  persons  and  the  rate  of  return  was  49%.  The  data 
obtained  are  presented  in  Annex  A. 

Survey  of  Enrolled  Students  and  Registered  Supervisors  (3rd  Survey). 
Duri ng  August  1979,  questionnaires  were  mailed  to  all  active  Army 
enrolled  students  and  registered  supervisors  and  a  large  sample  (about 
two  thirds)  of  the  enrolled  students  and  registered  supervisors  who 
were  members  of  the  National  Guard  or  Army  Reserve.  The  general  pur¬ 
pose  of  these  questionnaires  (Annex  A)  was  to  collect  a  variety  of 
data  bearing  on  acceptance  of  the  program's  delivery  system,  course 
content,  and  test  scoring  procedures.  Questionnaires  were  sent  to  153 
active  Arr^y  students,  103  active  Army  supervisors,  300  NG/USAR  stu¬ 
dents,  and  200  NG/USAR  supervisors.  The  rates  of  return  for  these 
four  groups  were  13,  28,  22,  and  29  percent,  respectively.  The  data 
obtained  are  discussed  later  in  this  report. 


B.  STUDENT  PROGRESS 

The  extent  to  which  students  actively  participate  in  a  training 
program  can  be  used  as  a  measure  of  field  acceptance  of  the  program. 
Special  procedures  had  been  developed  to  manage  the  central  admin¬ 
istration  of  the  SOJET  program  during  its  initial  implementation. 

This  was  necessary  because  much  of  the  data  collected  as  part  of  the 
SOJET  program  could  not  be  processed  using  I  PD  data  files  and  computer 
programs.  These  procedures  included  the  collection  and  storage  of 
training  data;  training  records  were  maintained  on  special  OPS/INTEL 
program  files.  These  records  were  analyzed  to  develop  data  on  such 
matters  as  rate  of  student  progress,  student  drop-out  rates,  course 
completion  rates,  and  the  ratio  of  enrollment  to  actual  starts  in  the 
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program.  The  findings  from  this  analysis  are  presented  in  Section  IV 
of  this  report. 


C.  ADMINISTRATIVE  FEASIBILITY/PROBLEMS 

One  of  the  purposes  of  this  study  was  to  evaluate  the  SOJET 
program  delivery  system,  to  identify  problems  with  it,  and  when 
feasible,  to  immediately  implement  solutions  to  these  problems. 
Problems  experienced  or  anticipated  by  unit  personnel,  especially 
registered  or  potential  course  supervisors,  were  identified  during  the 
field  visits  and  during  the  first  two  surveys. 

A  second  set  of  problems  investigated  were  those  experienced  by 
persons  responsible  for  the  central  management  of  the  SOJET  program. 
Appropriate  staff  members  of  the  Institute  for  Professional  Develop¬ 
ment  were  interviewed,  and  through  extensive  discussions,  a  number  of 
administrative  problems  were  identified. 

Solutions  to  some  of  these  problems  were  implemented  in  the 
summer  of  1979.  Solutions  to  other  problems  were  incorporated  in  pro¬ 
cedures  in  the  SOJET  Program  Implementation  Handbook  (Appendix  B). 


D.  COST  OF  SOJET  PROGRAM 

To  determine  the  cost  of  the  SOJET  program,  costs  for  adminis¬ 
tering  the  entire  SOJET  program  had  to  be  estimated.  These  estimates 
were  constructed  by  first  obtaining  cost  data  for  the  administrative 
elements  comprising  the  delivery  system  for  the  typical  correspondence 
program.  Most  of  these  estimates  had  already  been  developed  by  IPD 
and  were  described  on  recently  prepared  "Schedule  Xs."  These  schedu¬ 
les  show  the  work  elements  of  a  major  operation  and  provide  detailed 
cost  estimates  for  each  element  and  operation.  The  schedules  are  used 
for  budget  planning. 

Each  administrative  work  element  for  a  traditional  correspondence 
course  was  then  reviewed  and  cost,  personnel,  or  time  requirements 
were  adjusted  to  reflect  SOJET  program  procedures  as  described  in  the 
SOJET  Program  Implementation  Handbook  (Annex  B).  IPD  personnel 
familiar  with  current  SOJET  administrative  procedures  cooperated  in 
this  casting  exercise. 


E.  REVISION  OF  SOJET  DELIVERY  SYSTEM 

Throughout  the  first  nine  months  of  this  project,  there  was  a 
continuing  effort  to  identify  administrative  problems  that  could  be 
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revised  immediately.  Discussions  with  IPD  personnel  and  an  analysis 
of  interview  and  questionnaire  data  resulted  in  identification  of  a 
number  of  problem  areas.  These  areas  were  reviewed  with  IPD  and  ARI 
personnel  to  specify  the  problems  for  which  solutions  could  be  deve¬ 
loped  and  implemented  without  delay.  Most  of  these  revisions  involved 
simplifying  procedures  or  program  information  and  enrollment  material. 
These  revisions  were  made  by  the  contractor,  reviewed  by  IPD  and  ARI, 
and  then  implemented  during  August  1979. 


F.  FEEDBACK  SYSTEM  DESIGN 

The  present  management  system  for  Army  correspondence  courses 
produces  a  number  of  monthly  reports  of  use  to  program  managers.  How¬ 
ever,  with  one  exception,  a  subcourse  test  item  analysis  report,  IPD 
prepares  no  management  reports  for  the  specific  use  of  course  develo¬ 
pers.  One  of  the  primary  purposes  of  this  project  was  to  design  an 
improved  feedback  system  for  program  managers  and  especially  for  SOJET 
course  developers.  This  was  accomplished  as  follows.  First,  a  deter¬ 
mination  was  made  of  the  decisions  that  must  be  made  by  program  mana¬ 
gers  and  by  course  developers.  These  decisions  were  identified 
through  discussions  with  IPD  personnel  and  telephonic  discussions  with 
course  develoers  at  the  combat  arms  schools.  Contractor  personnel 
then  reviewed  each  decision  area  and  identified  the  major  infor¬ 
mational  inputs  required  to  make  a  decision.  Thirdly,  the  Contractor 
examined  IPD  management  reports  and  data  files,  and  identified  the 
information  IPD  currently  can  provide  in  support  of  the  major  deci¬ 
sions  which  must  be  made  by  program  managers  and  by  course  developers. 
During  this  activity  enrollment  forms  and  training  records  were 
analyzed  to  determine  the  data  captured  by  such  forms  and  records.  A 
comparison  of  information  requirements  with  current  information 
collecting/generating  capabilities  resulted  in  the  identification  of  a 
number  of  information  gaps— information  that  should  be  collected  or 
generated  in  order  to  provide  course  developers  with  a  sound  basis  for 
decision  making.  As  a  final  step,  proposed  changes  in  enrollment 
forms  were  developed,  and  a  number  of  new  types  of  feedback  reports  to 
course  developers  were  proposed.  These  are  discussed  in  Section  VI  of 
this  report  and  in  ANNEX  B  to  the  report. 


G.  DESIGN  OF  ADMINISTRATIVE  SYSTEM  COMPATIBLE  WITH  INSTRUCTIONAL 
MANAGEMENT  INFORMATION  SYSTEM  (IMIS) 

Correspondence  courses  currently  are  managed  by  the  U.S.  Army 
Training  and  Doctrine  Command  (TRADOC)  educational  system  for  non¬ 
resident  instruction  (TREDS-NRI).  This  system  is  somewhat  obsolete 
and  does  not  have  the  capability  to  handle  anticipated  correspondence 
course  requirements  in  the  1980  time  frame.  Consequently,  the  system 
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is  being  revised.  The  new  system,  currently  known  as  the  Instruc¬ 
tional  Management  Information  System  (IMIS),  will  be  available  within 
two  years.  This  new  system  will  be  capable  of  storing  considerably 
more  data  and  will  have  an  expanded  data  processing  capability  IPD  is 
scheduled  to  acquire  its  own  computer  facility  in  support  of  this  new 
system. 

Currently,  certain  portions  of  the  SOJET  program  are  being 
managed  "by  exception."  That  is,  special  computer-based  files  are 
used  to  store  and  process  certain  types  of  SOJET  program  data  which 
cannot  now  be  processed  by  the  TREDS-NRI  system.  One  purpose  of  this 
project  was  to  eliminate  the  need  for  those  special  programs  and  files 
the  goal  was  to  design  an  administrative  system  that  can  be  handled  by 
the  new  IMIS.  Since  IMIS  will  not  be  operational  for  about  two  years, 
its  capabilities  and  the  programs  that  will  be  used  with  it  had  to  be 
estimated  for  the  purposes  of  this  project.  The  final  design  of  the 
SOJET  administrative  system  was  adjusted  so  that  it  was  compatible 
with  the  estimated  IMIS  capabilities.  To  accomplish  this,  the  func¬ 
tional  requirements  of  the  SOJET  administrative  system  were  iden¬ 
tified,  then  reviewed  with  persons  responsible  for  development  of 
IMIS.  The  final  version  of  the  SOJET  administrative  system  thus  was 
designed  so  that  it  can  be  handled  by  IPD 1 s  future  educational  manage¬ 
ment  information  system— IMIS. 


IV.  DESCRIPTIVE  FINDINGS 


A.  PROGRAM  PUBLICITY  AND  ENROLLMENT 

Implementation  of  the  SOJET  program  began  early  in  1978.  To 
announce  the  availability  of  the  program,  a  short  description  was 
mailed  to  all  Army,  National  Guard,  and  U.S.  Army  Reserve  battalions. 
Also,  articles  announcing  that  the  program  was  available  were  published 
in  school  newspapers.  Application  for  enrollment  into  the  program 
began  to  be  accepted  during  May  1978. 

As  of  the  end  of  October  1979,  728  students  had  been  enrolled,  498 
of  whom  were  from  National  Guard  or  Reserve  units.  As  shown  in  Figure 
3,  the  rate  of  enrollment  for  active  Army  personnel  was  very  slow 
until  Janury  1979. 

On-site  interviews  conducted  at  state-side  Army  installations  during 
the  fall  and  early  winter  of  1978  indicated  that  the  publicity  had  not 
been  effective.  Over  half  of  those  persons  eligible  for  enrollment 
were  not  aware  of  the  program  (Table  B-8,  Annex  A).  As  the  result  of 
this  finding  and  the  low  enrollment  rate  of  active  Army  personnel,  an 
intensive  publicity  campaign  was  begun  in  December  1978.  Messages 
were  sent  to  all  battalion  commanders  announcing  the  availability  of 
the  program;  a  description  of  the  program  was  mailed  to  the  commanding 
officer  of  all  battalion  intelligence  (S2)  and  operations  (S3) 
sections;  additional  articles  about  the  program  were  prepared  for 
School  papers;  and  overseas  radio  spots  were  prepared. 

As  the  result  of  this  campaign,  enrollment  of  active  Army  per¬ 
sonnel  began  to  increase  and  currently  is  rising  at  the  rate  of  about 
25  students  per  month.  However,  even  after  this  additional  publicity, 
over  60%  of  the  persons  interviewed  during  February-March  1979  still 
were  not  aware  of  the  SOJET  OPS/INTEL  program  (Table  B-8). 

The  somewhat  unusual  publicity  campaign  was  undertaken  to 
encourage  enrollment  in  the  program  as  rapidly  as  possible,  so  that 
numbers  of  students  would  be  adequate  for  a  beginning  on  evaluation  of 
the  program.  In  the  intervening  months,  availability  of  the  SOJET 
operations  and  intelligence  courses  has  been  announced  in  appropriate 
School  catalogs  and  the  courses  have  been  referenced  in  appropriate 
Soldier's  Manuals.  These  catalogs  and  Soldier's  Manual  listings  should 
be  sufficient  to  maintain  the  rate  of  enrollment  at  its  current  level. 


B.  TARGET  AUDIENCE  VERSUS  STUDENT  PROFILE 

The  SOJET  program  is  u.,-sual  in  that  there  are  no  restrictions  on 
who  can  enroll  in  the  program.  However,  the  operations  and  Intel li- 
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Figure  3.  Cumulative  Humber  of  Accepted  Enrollment 


gence  courses  were  designed  primarily  for  enlisted  personnel  assigned 
to  combat  battalions.  The  duty-oriented  nature  of  the  course  material 
makes  the  courses  most  appropriate  for  persons  now  in  operations  or 
intelligence  duty  positions  or  aspiring  to  such  positions. 

The  characteristics  of  persons  enrolled  in  the  SOJET  program  are 
shown  in  Figure  3  and  in  Tables  4  through  7.1  Most  SOJET  program 
enrollees  (78%)  were  members  of  the  National  Guard  or  of  Reserve 
Units.  However,  from  March  through  October  1979  almost  equal  numbers 
of  active  Army  and  National  Guard/Reserve  students  enrolled  in  the 
Program. 

Eighty  percent  of  the  enrollees  held  a  current  operations  or  in¬ 
telligence  duty  assignment  (Table  4).  Moreover,  persons  currently 
assigned  to  operations  (S3)  duty  positions  tended  to  enroll  in  one  of 
the  eight  operations  courses  while  persons  currently  assigned  to  in¬ 
telligence  (S2)  duty  positions  tended  to  enroll  in  one  of  the  eight 
intelligence  courses  (Table  4).  Also,  persons  currently  assigned  to  a 
sergeant  or  assistant  sergeant  duty  position  usually  enrolled  for  one 
of  the  eight  senior  courses  while  persons  currently  assigned  to 
assistant  or  specialist  positions  enrolled  in  one  of  the  eight  junior 
courses  (Table  4).  Most  of  the  persons  currently  assigned  to  non¬ 
operations  or  intelligence  positions  enrolled  in  one  of  the  operations 
sergeant  courses. 

Seventh-three  percent  of  all  enrollees  were  NCOs  at  the  3,  4  or  5 
skill  level  (Table  5).  This  seems  to  reflect  the  fact  that  for  a 
number  of  MOSs  persons  have  to  obtain  a  fairly  senior  rank  before 
there  is  a  requirement  for  them  to  learn  $2  or  S3-related  tasks. 

Eighty-seven  percent  of  the  students  held  a  combat  MOS  (Table  6). 
The  largest  group  of  enrollees  (30%)  held  an  infantry  MOS  while  an 
additional  22%  held  a  field  artillery  MOS.  Most  students  who  held  a 
non-combat  MOS  held  an  MOS  in  the  intelligence  area. 


^Detailed  versions  of  these  tables  are  located  in  Appendix  A. 
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TABLE  4 


DISTRIBUTION  OF  ENROLLMENTS  IN  TERMS  OF  CURRENT  DUTY  POSITION3 


Course 

Component 

OPS  SGT/ 
ASST  OPS 
SGT 

CURRENT  DUTY  POSITION 

INTEL  SGT/  OPS  INTEL 

ASST  INTEL  ASST/  ASST/ 

SGT  SPEC  SPEC 

NON  OPS/ 
INTEL 
POSITIONS 

Total 

OPS  SGT 

AA 

54 

2 

. 

36 

92 

(Y01-Y04) 

NG/USAR 

148 

11 

4 

- 

46 

209 

INTEL  SGT 

AA 

1 

24 

» 

1 

9 

35 

(Y05-Y08) 

NG/USAR 

4 

82 

6 

1 

0 

99 

OPS  SPEC 

AA 

1 

1 

23 

3 

3 

31 

(Y09-Y12) 

NG/USAR 

2 

- 

35 

8 

9 

54 

INTEL  SPEC 

AA 

_ 

1 

7 

5 

13 

(Y13-Y16) 

Selected 

NG/USAR 

4 

3 

19 

3 

29 

Sub- 

AA 

12 

8 

2 

1 

4 

27 

Courses^ 

(Y01-Y16) 

NG/USAR 

20 

11 

1 

9 

4 

45 

Total 

AA 

68 

35 

26 

12 

57 

H 

NG/USAR 

174 

108 

49 

37 

68 

m 

3Enrollment  as  of  1  September  1979 

^Persons  who  enrolled  in  less  than  a  full  course 
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TABLE  5 


DISTRIBUTION  OF  ENROLLMENTS  IN  TERMS  OF  CURRENT  SKILL  LEVEL3 


Course 

Component 

1 

Current 

2 

Skill 

3 

Level 

4 

5 

Unknown 

Total 

OPS  SGT 

AA 

3 

11 

19 

50 

7 

2 

92 

(Y01-Y04) 

NG/USAR  ! 

3 

7 

34 

70 

88 

7 

209 

INTEL  SGT 

AA 

1 

5 

15 

9 

2 

3 

35 

(Y05-Y08) 

NG/USAR 

- 

3 

23 

32 

39 

2 

99 

OPS  SPEC 

AA 

25 

5 

1 

_ 

31 

(Y09-Y12) 

NG/USAR 

23 

12 

11 

2 

1 

5 

54 

INTEL  SPEC 

AA 

10 

1 

2 

_ 

13 

(Y13-Y16) 

NG/USAR 

6 

14 

8 

1 

- 

- 

29 

Selected 

AA  | 

6 

6 

4 

3 

5 

3 

27 

Sub- 

Courses^ 

NG/USAR 

3 

5 

3 

13 

20 

1 

45 

Total 

AA 

45 

28 

41 

62 

14 

8 

11 

NG/USAR 

35 

41 

79 

118 

148 

15 

aEnrollment  as  of  1  Sepiember  1979 


^Persons  who  enrolled  in  less  than  a  full  course 
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TABLE  6 


DISTRIBUTION  OF  ENROLLMENTS  IN  TERMS  OF  CURRENT  MOSa 


Course 

Component 

AR 

Combat  MQSC 
INF  FA 

ADA 

ENG 

Non-Combat  M0Sc 
INTEL  ORD 

OTHER 

UNK 

Total 

OPS  SGT 

AA 

9 

31 

36 

10 

3 

. 

. 

i 

2 

92 

(Y01-Y04) 

NG/USAR 

46 

71 

42 

5 

7 

7 

11 

15 

5 

209 

INTEL  SGT 

AA 

3 

15 

4 

5 

6 

2 

• 

* 

35 

(Y05-Y08) 

NG/USAR 

38 

22 

22 

1 

- 

12 

2 

1 

1 

99 

OPS  SPEC 

AA 

10 

2 

10 

4 

2 

3  | 

31 

(Y09-Y12) 

NG/USAR 

14 

20 

6 

3 

1 

2 

2 

2 

4 

54 

INTEL  SPEC 

AA 

_ 

_ 

2 

4 

6 

* 

_ 

1 

13 

(Y13-Y16) 

NG/USAR 

2 

4 

4 

1 

- 

16 

- 

2 

- 

29 

Selected 

AA 

. 

13 

3 

8 

1 

_ 

2 

_ 

27 

Sub- 

Courses^ 

NG/USAR 

6 

7 

19 

2 

1 

1 

2 

7 

45 

Total 

AA 

12 

69 

47 

37 

9 

13 

2 

6 

m 

124 

93 

12 

9 

38 

17 

27 

Enrollment  as  of  1  September  1979 

^Persons  who  enrolled  in  less  than  a  full  course 


CAR  *  Armor,  INF  *  Infantry,  FA  «  Field  Artillery,  ADA  ■  Air  Defense  Artillery, 
ENG  *  Engineering,  INTEL  *  Intelligence,  and  ORD  «  Ordnance 


As  expected,  many  students  had  been  in  their  current  duty  posi¬ 
tion  for  a  short  period  of  time.  Thirty-nine  percent  and  twenty-nine 
percent  of  active  Arn^y  and  NG/USAR  students,  respectively,  had  held 
their  present  duty  position  for  six  moths  or  less  (Table  7).  On  the 
other  hand,  78%  and  82%,  respectively,  of  active  Amy  and  NG/USAR 
students  had  a  total  of  seven  or  more  months  in  all  operations  or 
intelligence  assignments.  This  finding  suggests  that  most  students 
were  fairly  familiar  with  operations  or  intelligence  tasks. 

The  foregoing  profile  data  suggest  that  the  SOJET  program  was 
reaching  an  appropriate  target  audience.  The  typical  SOJET  student 
was:  a  senior  NCO,  enrolled  in  a  course  appropriate  to  his  duty  posi¬ 
tion,  held  a  combat  MOS,  and  had  less  than  12  months  of  experience  in 
his  present  duty  position.  In  recent  months  about  50%  of  the  students 
were  members  of  the  active  Arn\y. 


C.  STUDENT  PROGRESS 

As  of  1  September  1979  six  hundred  and  thirty-four  (634)  students 
had  been  accepted  into  the  SOJET  program.  The  training  status  of 
these  students  as  of  21  September  1979  is  shown  in  Table  8.  In  addi¬ 
tion  to  showing  the  training  status  of  Active  Army  (AA)  and  National 
Guard/USAR  students,  this  table  shows  the  training  status  of  students 
who  enrolled  at  different  time  intervals.  These  groups  will  be  called 
“classes".  Class  1  consists  of  students  who  enrolled  during  May-June 
1978.  Class  2  students  enrolled  during  July-August  1978.  Students 
who  enrolled  during  July-August  1979  are  referred  to  as  "Class  8". 
Percentages  calculated  on  the  basis  of  total  enrollment  show  that 
program  completion  rates  were  low  for  both  active  Army  (6%)  and 
NG/USAR  (10%)  students.  Program  completion  rates  did  improve  somewhat 
for  those  classes  of  students  which  have  been  in  the  program  a  longer 
period  of  time.  Table  8  shows  also  that  many  students  had  never  sub¬ 
mitted  a  test  ever  though  they  had  been  enrolled  in  the  program  for  a 
considerable  length  of  time.  It  should  be  noted  also  that  par¬ 
ticipation  by  National  Guard  and  USAR  students  was  greater  than  for 
active  Army  students  both  in  terms  of  the  actual  number  of  persons  who 
enrolled  ir  the  program  (437  versus  198)  and  the  percent  of  students 
who  were  active  to  some  degree  in  the  program  (49  %  versus  30%). 

Table  9  shows  the  percent  of  students  active  during  various  time 
periods  following  enrollment.  For  this  table,  "active"  is  defined  as 
the  submission  of  a  test  for  a  new  subcourse;  the  test  may  be  a 
passing  or  failing  one.  The  numbers  displayed  in  Table  9  indicate 
that  a  lower  percent  of  active  Army  students  submitted  subcourse  tests 
than  did  NG/USAR  students. 

As  noted  earlier,  most  students  enrolled  in  either  the  OPS  SGT  or 
the  INTEL  SGT  courses.  For  these  two  sets  of  courses  Table  9  shows 
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TABLE  7 


DISTRIBUTION  OF  ENROLLMENTS  IN  TERMS  OF  MONTHS  IN  CURRENT 
DUTY  POSITION,  AND  MONTHS  IN  ALL  OPERATIONS  OR  INTELLIGENCE 

DUTY  ASSIGNMENTS 


Duty  Time 
in  Months 

Months  irr  Current  Position 

AA  NG/USAR 

(N  «  198)  (N  »  436) 

Months  in  All  OPS/INTEL 
Duty  Assignments 

AA  NG/USAR 

(N  -  198)  (N  -  436) 

1  -  6 

39% 

29% 

22% 

18% 

7  -  12 

28% 

24% 

22% 

17% 

13  -  18 

16% 

12% 

13% 

10% 

19  -  24 

9% 

9% 

7% 

8% 

25  -  36 

4% 

9% 

12% 

11% 

37  -  48 

- 

4% 

4% 

8% 

49+ 

- 

7% 

12% 

21% 

Unknown 

4% 

6% 

8% 

7% 

PERCENT  OF  STUDENTS  ACTIVE  DURING  VARIOUS  TIKE  PERIODS  FOLLOWING  ENROLLMENT3 
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that  the  highest  percent  of  students  were  active  during  the  first  two 
months  following  enrollment.  This  suggests  that  most  students,  if 
they  are  going  to  participate  at  all  in  the  program,  begin  their  par¬ 
ticipation  soon  after  receiving  their  lesson  material.  Table  9  shows 
also  that  after  6  months  participation  in  the  program  drops  to  eight 
percent  for  AA  students;  for  NG/USAR  students  this  low  level  of  par¬ 
ticipation  is  not  reached  until  12  months  after  enrollment.  To  anti¬ 
cipate  the  next  section  of  this  report,  this  last  finding  seems  to  be 
due  to  personnel  turbulence  within  the  active  Army. 

Table  10  shows  the  cumulative  rate  at  which  new  subcourse  tests 
were  submitted  following  various  periods  after  enrollment.  This  table 
is  based  on  students  who  submitted  at  least  one  test,  passing  or 
failing.  The  data  reveal  that  active  Army  students  submitted  many 
more  tests  than  did  NG/USAR  students  during  the  early  part  of  their 
participation  in  the  SOJET  program.  At  the  end  of  six  months  the 
average  active  Arny  student  had  submitted  tests  over  approximately 
12  different  subcourses  as  compared  with  six  subcourses  for  NG/USAR 
students.  For  operations  and  intelligence  assistant/specialist  cour¬ 
ses  this  relationship  held  to  a  much  lesser  degree.  For  those  stu¬ 
dents  enrolled  in  selected  courses  only,  the  relationship  was 
reversed— more  tests  were  submitted  by  NG/USAR  students  than  by  active 
Arny  students. 

The  data  contained  in  Tables  9  and  10  seemed  to  reflect,  in 
general,  the  opportunities  for  self-study  which  exist  within  the 
active  Army  and  within  the  National  Guard/Reserves.  Within  the  active 
Arny  a  student  can  look  forward  to  reassignment  to  a  new  duty  position 
within  a  few  months.  Therefore,  if  he  is  to  participate  in  a  program 
he  must  begin  his  participation  as  soon  as  possible  and  proceed  as 
rapidly  as  he  can  through  his  program  of  study.  On  the  other  hand, 
members  of  national  guard  and  reserve  units  are  most  apt  to  par¬ 
ticipate  in  training  programs  as  part  of  their  schedule  of  monthly 
training  activities.  The  data  in  Table  10  suggests  also  that  students 
enrolled  in  the  SOJET  Program  should  be  given  longer  than  12  months  to 
complete  their  program.  This  appears  to  be  especially  true  for 
national  guard  and  reserve  students. 

The  SOJET  program  places  no  restrictions  on  the  number  of  sub¬ 
courses  a  student  can  enroll  for.  A  few  students  enroll  for  only  one 
or  two  subcourses  while  a  few  others  enroll  for  30  or  more  subcourses. 
About  25%  of  the  students  will  enroll  for  one  of  the  16  full  courses 
plus  a  few  additional  subcourses  of  interest  to  them.  For  some  stu¬ 
dents,  therefore,  the  submission  of  tests  over  four  different  subcour¬ 
ses  might  mean  that  they  have  completed  their  total  program  of 
instruction.  For  other  students  a  submission  of  this  number  of  tests 
may  mean  that  they  have  completed  only  10  to  20%  of  their  program. 
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Table  11  shows  the  percent  of  program  completed  following  various 
months  after  enrollment.  Each  percent  figure  contained  in  Table  11  is 
based  on  a  different  number  of  students,  namely  the  number  of  active 
students  still  remaining  in  the  program.  Therefore,  those  percentages 
listed  under  intervals  13-14  and  15-16  (classes  1  and  2,  respectively) 
•are  based  on  a  very  small  number  of  students.  The  percentages 
displayed  in  Table  11  reflect  the  findings  already  discussed  with 
respect  to  Table  10.  After  six  months  all  participating  active  AA 
students  had  completed  57%-of  their  program  versus  34%  for  NG/USAR. 

Collectively  the  data  displayed  in  Tables  7  through  11  indicate 
that  based  on  jHJL  enrol  lees:  (1)  only  about  50%  can  be  expected  to 
actively  participate  in  their  program  of  instruction.  The  remainder 
will  become  "no-starts";  and  (2)  the  percent  of  persons  completing 
their  entire  program  will  be  quite  low,  on  the  order  of  10%. 

Based  on  students  who  are  active  at  some  point  in  the  program, 
student  progress  data  suggests  that  (1)  most  students  are  active  early 
in  their  program  (Table  9);  (2)  most  new  subcourse  tests  will  be 
submitted  during  the  first  six  months  of  the  program  (Table  10); 

(3)  the  number  of  different  tests  submitted  will  be  much  higher  for  AA 
than  for  NG/USAR  students  (Table  10),  and  (4)  of  those  students  who 
participate  at  all  in  a  program  most  of  them  will  complete  a  high  per¬ 
centage  of  their  instructional  program  (Table  11)."  It  seems  reaso¬ 
nable  to  conclude  that  these  people  will  derive  considerable  benefit 
from  participating  in  the  program. 


D.  IMPACT  OF  PERSONNEL  TURBULENCE 

In  the  S0JET  program  the  requirement  that  the  course  supervisor 
administer  and  score  subcourse  examinations  inevitably  sets  the  stage 
for  disruptions  in  student  progress  whenever  the  course  supervisor  is 
unavailable.  Temporary  unavailability  may  be  caused  by  the  supervisor 
being  in  the  field,  on  TOY,  or  on  leave.  When  the  supervisor  is 
reassigned  to  another  job,  undergoes  a  permanent  change  of  station,  or 
is  discharged,  the  student  will  have  to  obtain  a  new  course  super¬ 
visor.  Likewise,  when  the  student  is  reassigned  or  sent  to  a  new 
post,  his  new  duties  are  very  likely  to  conflict  with,  and  may  not 
even  be  relevant  to,  the  S0JET  course  in  which  he  is  enrolled.  Thus, 
he  would  have  little  incentive  for  continuing  in  the  program;  at  a 
minimum  he  would  have  to  arrange  for  a  new  course  supervisor. 

According  to  IPD  estimates,  personnel  turnover  within  the  active 
Army  is  about  25%  per  quarter.  Based  on  information  acquired  from 
active  Army  personnel  during  the  field  visits  conducted  as  part  of 
this  study,  it  is  estimated  that  the  annual  turnover  rate  for  enlisted 
and  for  officer  personnel  is  at  least  50%  and  100%,  respectively.  In 
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view  of  such  estimates,  there  is  a  high  probability  that  many  of  the 
potential  “no-starts"  and  "drop-outs"  reflected  in  the  data  presented  in 
Table  8  are  due  to  personnel  turnover. 

As  a  prelude  to  the  third  survey  of  this  study,  an  effort  was 
made  to  contact  by  telephone  all  active  Army  course  supervisors 
assigned  to  a  CONUS  installation.  From  these  telephone  contacts,  spe¬ 
cific  information  was  obtained  on  the  present  location  of  104  of  111 
enrolled  students  and  61  of  65  registered  supervisors.  This  infor¬ 
mation  is  shown  in  Table  12. 

Of  the  104  students,  73  still  were  assigned  to  the  unit  they  were 
in  when  they  first  enrolled  in  the  SOJET  program;  however,  8  of  the  73 
were  in  the  field,  on  TDY,  or  on  leave.  The  remaining  31  persons  were 
no  longer  in  their  original  unit;  most  of  them  having  been  reassigned 
to  another  installation.  Thus,  it  would  appear  that  reassignment 
accounted  for  lack  of  participation  by  about  30%  of  the  students. 

Of  the  61  course  supervisors,  26  (43%)  were  no  longer  available 
to  carry  out  their  responsibilities  as  course  supervisors;  12  had  been 
reassigned  to  another  job,  11  had  undergone  a  permanent  change  of  sta¬ 
tion,  and  3  had  been  discharged. 

In  the  SOJET  program  unavailability  of  either  a  student  or  a 
supervisor  can  disrupt  student  progress.  From  the  telephone  survey 
described,  the  status  of  111  student-supervisor  pairs  was  determined. 
These  data  are  displayed  in  Table  13.  Positive  information  was 
obtained  on  100  student-supervisor  pairs;  for  the  other  11  pairs,  the 
status  of  one  or  both  parties  was  unknown.  In  those  100  pairs  for 
which  information  was  available,  there  were  47  pairs  in  which  both  the 
student  and  the  supervisor  were  still  assigned  to  the  same  unit  they 
were  in  when  they  first  enrol led/»egistered  in  the  program.  Although 
some  of  these  people  were  on  TDY  or  on  leave,  they  were  judged  as 
being  available  to  continue  in  the  SOJET  program.  For  the  remaining 
53  pairs,  either  the  supervisor,  the  student,  or  both  were  away 
attending  school,  had  been  assigned  to  another  job,  had  been  assigned 
to  another  post,  or  had  left  the  service.  On  the  basis  of  the 
foregoing  information,  it  is  estimated  that  at  least  50%  of  the  stu¬ 
dents  who  enroll  in  the  SOJET  program  and  who  are  members  of  the 
active  Army  will  have  their  program  disrupted. 

When  a  student  is  reassigned  to  a  new  job  or  undergoes  a  per¬ 
manent  change  of  station,  he  is  temporarily  disenrolled  from  the  SOJET 
program.  Once  he  is  established  in  his  new  job  or  station,  he  has  the 
option  of  re-enrolling  in  the  program  and  continuing  his  course  of 
study.  This  option  is  seldom  exercised,  probably  because  personnel 
often  are  temporarily  assigned  to  in  the  $2  and  S3  sections  of  a  bat¬ 
talion  until  they  are  needed  elsewhere;  when  they  are  reassigned,  they 
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TABLE  12 


STATUS  OF  ACTIVE  ARMY  STUDENTS  AND  COURSE 
SUPERVISORS  (CONUS)  AS  OF  AUGUST  1979 


Status 

Enrolled 

Students 
(N  =  111) 

Registered 

Supervisors 
(N  =  65) 

In  Unit 

65 

25 

In  Field 

5 

4 

On  Leave 

1 

3 

TDY 

2 

3 

Reassigned 

6 

12 

PCS 

24 

11 

Discharged 

1 

3 

Unknown 

7 

4 

TABLE  13 

STATUS  OF  STUDENT-SUPERVISOR  PAIRS 
AS  OF  AUGUST  1979* 

(N  »  111) 


Supervisors 

Students 

imimna 

Not  Available 

Unknown 

Available 

47 

17 

1 

Not  Available 

22 

14 

- 

Unknown 

4 

— 

6 

Total 

73 

31 

7 

aStatus  as  of  20  August  1979 
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have  little  reason  to  continue  their  S2  or  S3  course  of  instruction 
unless  they  have  an  M0$  in  which  operations  or  intelligence  tasks  are 
covered  in  their  Soldier's  Manual. 

The  data  presented  in  Table  13  suggest  that  there  is  a  rela¬ 
tionship  between  the  number  of  months  since  enrollment  and  turnover 
rates.  Table  14  contains  data  bearing  on  this  relationship. 
Examination  shows  that  for  the  47  pairs  where  both  the  supervisor  and 
student  were  available,  it"  had  been  an  average  of  six  months  since 
enrollment  in  the  SOJET  program.  For  the  22  pairs  in  which  the  super¬ 
visor  was  gone  but  the  student  was  still  in  the  unit,  the  students  had 
enrolled  on  an  average  of  seven  months  earlier.  For  17  additional 
student-supervisor  pairs,  the  student  was  gone  but  the  supervisor  was 
present;  those  pairs  had  enrolled  in  the  program  approximately  eight 
months  earlier.  Finally,  for  those  14  pairs  where  both  the  student 
and  supervisor  were  gone  from  the  unit,  those  pairs  had  enrolled  an 
average  of  10  months  prior  to  the  telephone  survey.  The  foregoing 
data  substantiate  the  obvious— the  longer  the  period  since  enrollment 
into  the  SOJET  program,  the  higher  the  probability  that  both  the  stu¬ 
dent  and  supervisor  will  no  longer  be  assigned  to  the  unit  they  were 
in  when  they  first  enrolled/registered  into  the  program. 

Another  set  of  data  bearing  on  personnel  turbulence  is  displayed 
in  Table  15.  Three  questions  on  personnel  turnover  from  the  super¬ 
visor  questionnaire  used  for  the  third  survey,  are  listed  in  Table  15, 
along  with  the  percentage  of  persons  who  provided  various  responses 
to  each  question.  In  58%  or  more  of  the  cases,  the  supervisors  stated 
that  personnel  turnover  did  not  prevent  students  from  enrolling  in  the 
program,  prevent  students  from  completing  the  program,  or  make  super¬ 
visors  reluctant  to  register  in  a  program.  This  data  supports  the 
telephone  survey  findings— personnel  turbulence  may  prevent  30-40% 
of  enrolled  students  from  completing  this  program.  It  also  may  pre¬ 
vent  considerable  numbers  of  potential  students  from  enrolling  in  the 
program. 

The  impact  of  personnel  turbulence  on  test  submission  rates  also 
can  be  inferred  from  the  data  presented  in  Tables  9  and  10.  Table  10 
shows  that  the  rate  of  test  submission  for  active  Am\y  personnel  was 
more  rapid  than  that  for  N6/USAR  personnel  through  the  first  four 
months  of  their  enrollment.  After  six  months,  very  few  active  Amy 
students  continued  to  submit  tests  while  many  NG/USAR  students  con¬ 
tinued  to  do  so.  Based  on  the  foregoing  findings  it  seems  highly  pro¬ 
bable  that  personnel  turbulence  had  an  adverse  effect  on  student 
progress,  especially  within  the  active  Am\y.  Removing  the  require¬ 
ment  for  a  registered  course  supervisor  might  decrease  this  effect 
somewhat. 
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TABLE  14 


PERSONNEL  TURBULENCE  OF  STUDENT-SUPERVISOR  PAIRS 
AS  RELATED  TO  MONTHS  SINCE  ENROLLMENT 


Status  of  Student-Supervisor 

Pairs 

N 

Average  Number  of  Months 

Since  Enrollment 

Both  student  and  supervisor  present 

47 

6  months 

Supervisor  gone,  student  present 

22 

7  months 

Student  gone,  supervisor  present 

17 

8  months 

Both  student  and  supervisor  gone 

14 

10  months 

TABLE  15 

SUPERVISOR  REPLIES  TO  QUESTIONS  ABOUT 
PERSONNEL  TURNOVER* 


Response  Options 

Question 

Component 

N 

m 

Has  personnel  turnover: 

a.  Prevented  potential  students 

AA 

29 

4% 

from  enrolling  in  the  program? 

NG/USAR 

53 

13% 

27% 

m 

b.  Prevented  stgudents  from 

AA 

29 

8% 

33% 

58% 

completing  program? 

NG/USAR 

53 

9% 

22% 

69% 

c.  Made  you  or  others  reluctant 

AA 

29 

21% 

21% 

58% 

to  register  as  a  course 
supervisor? 

NG/USAR 

53 

11% 

13% 

76% 

aThird  Survey  Data 
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E.  COST  COMPARISON  OF  SOJET  AND  IPD  CENTRAL  MANAGEMENT  PROCEDURES 

During  this  effort  the  absolute  and  comparative  costs  of  admi¬ 
nistering  a  correspondence  course  by  current  IPD  and  by  SOJET  original 
and  revised  procedures  were  calculated.  As  a  first  step,  the  time 
•needed  to  perform  the  tasks  required  of  each  IPD  staff  member  was 
determined.  A  recent  "Schedule  X"  (Workload  Estimate  Chart)  prepared 
by  IPD  provided  the  task  listings  and  time  estimates  for  IPD  proce¬ 
dures.  These  procedures  were  reviewed  with  IPD  personnel  and  the  time 
estimates  for  each  task  raised  or  lowered  to  reflect  SOJET  program 
procedures.  Whenever  possible,  time  estimates  for  tasks  were  based  on 
actual  work  counts  and/or  on  personnel  observation.  All  time  and  non¬ 
labor  cost  estimates  were  derived  so  that  they  could  be  converted  to 
"cost  per  enrolled  student  per  year." 

The  entries  in  Table  16  illustrate  how  time  requirements  were 
calculated.  IPD  enrollment  clerks  are  responsible  for  processing 
enrollment  applications,  a  task  that  can  be  divided  into  five  ele¬ 
ments.  Listed  under  IPD  procedures  (Table  16)  are  the  estimates  of 
the  time  required  to  perform  each  task  element  of  the  enrollment  task. 
In  the  SOJET  program,  the  original  procedures  involved  processing  two 
more  enrollment  forms  than  did  IPD  procedures,  and  one  of  these  forms, 
the  Student  Training  Plan,  was  time  consuming  to  edit  and  code.  On 
the  other  hand,  the  SOJET  program  has  no  eligibility  requirements  or 
equivalent  credits,  eliminating  two  steps  in  the  enrollment  process 
that  must  be  followed  under  IPD  procedures. 

Table  16  shows  that,  following  IPD  procedures,  it  takes  8.25 
minutes  to  process  one  enrollment  application.  The  process  requires 
9.25  minutes  per  student  when  the  original  SOJET  enrollment  proce¬ 
dures  are  used.  Using  revised  SOJET  procedures,  it  takes  about  7.5 
minutes  to  enroll  one  student-supervisor  pair. 

To  prepare  estimates  of  annual  manpower  requirements,  IPD  con¬ 
verts  "minutes  to  perform  a  task"  into  hours  per  year  required  to 
handle  a  full  student  complement.  From  July  1978  through  June  1979, 
IPD  processed  202,009  enrollment  forms  (DA  Form  145).  Thus,  using  IPD 
enrollment  procedures  an  estimated  27,776  hours  (202,009  x  8.25  minu¬ 
tes  ▼  60)  were  required  in  that  year  to  process  student  enrollments. 

Using  procedures  similar  to  those  just  described,  IPD  has  calcu¬ 
lated  the  hours  required  by  each  of  its  organizational  elements  to 
support  202,009  students  (Table  17).  The  estimates  were  adjusted  to 
reflect  original  and  revised  SOJET  procedures.  Of  particular  interest 
is  the  estimate  that,  for  IPD  and  for  revised  SOJET  procedures,  the 
time  required  to  process  and  to  support  one  student  is  approximately 
1.3  and  1.75  hours,  respectively.  Thus,  revised  SOJET  procedures 
require  almost  36%  more  labor  than  do  current  IPD  procedures. 
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TABLE  16 


TIME  REQUIRED  TO  PERFORM  ENROLLMENT  TASK  IN  ACCORDANCE  WITH  IPD 
PROCEDURES,  ORIGINAL  SOJET  PROCEDURES 
AND  REVISED  SOJET  PROCEDURES 
(MINUTES) 


Central  Management  Procedures 

TASK 

IPDa 

Standard 

SOJET 

(Original) 

SOJET 

(Revised) 

Student  Supervisor 

Student  Supervls 

Process  Incoming  Enroll¬ 
ment  Applications 

a.  Review  forms  for 
correctness/ 
completeness 

0.25 

0.50  0.25 

0.25  0.25 

b.  Compare  request  with 
catalog  entry  to 
determine  eligibility/ 
availability 

3.00 

c.  Check  for  and  certify 
equivalent  credits 
for  student 

1.50 

.  . 

d.  Code  information  for 
computer  input 

2.00 

5.00  2.00 

3.00  1.00 

e.  Make  up  and  mail 
welcome  letter 

1.50 

1.50 

1.50  1.50 

Total  Minutes 

8.25 

5715 

7.50 

aTask  elements  and  time  requirements  under  IPD  procedures  based  on  data 
developed  by  Institute  for  Professional  Development. 


TABLE  17 


COMPARISON  OF  LABOR  REQUIREMENTS  AND  PRINTING  AND  MAILING  COSTS 
REQUIRED  TO  ADMINISTER  ENTIRE  ACCP  ACCORDING  TO  IPD, 
ORIGINAL  SOJET  AND  REVISED  SOJET  PROCEDURES 


I  Central  Management  Procedures 

Cost  Categories 

IPD 

Standard 

SOJET 

(Original) 

SOJET 

(Revised) 

Labor  Costs 

Student  Services  Division 

a.  Office  of  Division  Chief 

b.  TREDS-NRI  Section 

c.  Processing  Branch 

d.  Administrative  Branch 

2,788  hrs. 
6,800 
153,637 
12,444 

2,788  hrs. 
6,816 
356,171 

18,300 

2,788  hrs. 
6,834 

161,371 

14,292 

Production  and  Distribu¬ 
tion  Division 

83,631 

167,262 

167,262 

Total  Man-hour 

Requirements 

259,300 

551,377 

352,547 

Man-hours  Required  per 
Student  (N=202,009 
Students) 

1.284 

2-729 

1-745 

Salary  Required  to  Pro¬ 
cess  One  Student  At 
Average  of  $6. 00/Hr. 

$  7.70 

$16.33 

$10.47 

Non-Labor  Costs 

Printing:  Cost  to  support 
one  student3 

Mailing:  Cost  to  support 
one  student a*b 

$18.12 

7.38 

$20.92 

16.48 

$18.60 

14.64 

Estimated  tost  to  Process 
and  Support  One  Student 

$33.20 

$53.73 

$43.68 

^Assume  that  IPD  and  SOJET  students  are  each  sent  an  average  of  12  subcourses. 

^Suheourses  sent  by  book  rate;  other  material  sent  by  First  Class 
Mailing  costs  based  on  non-discounted  rates. 
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SOJET  procedures  are  more  labor-intensive  than  those  used  by  IPD 
in  part  because  a  supervisor  must  be  registered  along  with  each 
student.  This  doubles  the  paperwork  required  to  enroll  one  student. 
Also,  SOJET  procedures  require  that  subcourse  material  be  sent  to  both 
the  student  and  the  course  supervisor,  a  requirement  that  contributes 
to  the  doubling  of  mailing  costs  and  the  labor  associated  with  these 
activities.  An  examination  of  Table  18  will  clarify  these  points. 


Table  18  lists  the  tasks  on  which  IPD  and  SOJET  labor  require¬ 
ments  differ  by  at  least  10%.  For  each  listed  task,  the  annual  hours 
required  to  process  202,009  students  were  determined.  Points  to  note 
are: 

o  For  11  of  16  listed  activities,  manpower  requirements  are 
higher  for  SOJET  procedures. 

o  Mailing  out  a  SOJET  course  guide  (original  procedures)  is 
the  most  time-consuming  requirement  of  the  SOJET  program. 
About  10  minutes  per  guide  are  required.  Under  the  re¬ 
vised  procedures  the  course  guide  would  be  discontinued. 

o  Mailing  test  material  separately  to  supervisors  doubles  the 
amount  of  labor  required  to  distribute  subcourse  material. 

o  The  requirement  to  register  the  supervisor  greatly  increases 
the  time  needed  to  process  source  documents,  maintain 
quality  control  over  source  documents,  and  complete  work 
unit  logs. 

The  figures  in  Table  17  demonstrate  that  the  central  management 
of  correspondence  courses,  even  though  computer  supported,  is  a  highly 
labor-intensive  operation  because  of  the  number  of  times  different 
pieces  of  paper  must  be  handled.  For  example,  a  one-minute  increase  in 
tlu*  time  required  to  process  200,000  enrollment  applications  would 
increase  IPD's  workload  by  3,333  hours.  Obviously,  doubling  the 
number  of  forms  in  a  student  application  packet  would  increase  IPD's 
annual  workload  by  thousands  of  hours. 
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TABLC  18 


ESTIMATED  ANNUAL  MANHOURS  REQUIRED  FOR  SELECTED  ACCP  ACTIVITIES 
BASED  ON  IPD,  ORIGINAL  SOJET,  AND  REVISED  SOJET  PROCEDURES  (HOURS) 


■BUSH 

Management  Procedures 

Organizational  Element  and  Work 

Activity 

IPD 

Standard 

SOJET 
(Original ) 

SOJET 

(Revised) 

Student  Services  Division 

a.  Office  of  Division  Chief 

- 

- 

- 

b.  TREDS-NRI  Section 

- 

- 

- 

c.  Processing  Branch 

(1)  Branch  Chief 

• 

- 

- 

(2)  Team  Chief 

(a)  Breakdov/n  work  for  team  daily 

313 

939 

626 

(b)  Decision  making/ interpreting 

policy 

247 

210 

210 

(c)  Maintain  records 

65 

78 

76 

(3)  Sr.  Training  Technician 

(a)  Correct  edit  listing 

173 

47 

31 

(b)  Quality  control  of  source 

documents 

4,230 

11,421 

7,614 

(c)  Determine  waiverabil  ity 

520 

1,040 

1,040 

(d)  Review  Form  Letter  IS 

867 

3,466 

216 

(4)  Training  Technician 

(a)  Process  source  documents 

4,506 

12,797 

8,111 

(b)  Completion  of  work  unit  log 

2,050 

9,082 

6,888 

(c)  Process  supervisor  changes 

- 

3,872 

3,872 

(5)  Enrollment  Clerk 

(a)  Process  Enrollment  forms 

27,776 

31,143 

25,251 

(b)  Return  erroneous  applications 

2,525 

7,575 

3,367 

(c)  Mail  SOJET  Course  Guide  , 

• 

30,4n 

- 

(d)  Complete  work  unit  log 

2,050 

1,304 

1,304 

d.  Administration  Branch 
(1)  All  personnel 

(a)  OptMiing/processing  mail 

12,444 

18,300 

14,292 

Production  and  Distribution  Division 
a.  All  personnnel 

(1)  All  required  activities 

J£L6.?.L 

_167t?6?_ 

167,262 

30 


The  increased  cost  of  the  SOJET  program  can  be  attributed  pri¬ 
marily  to  the  need  for  subcourse  test  scoring  guides.  This  is  a 
document  not  normally  employed  with  correspondence  courses.  In  addi¬ 
tion  to  preparation  costs,  scoring  guides  must  be  printed,  stored, 
retrieved  and  mailed.  Any  training  program  that  employs  test  scoring 
guides  usually  will  incur  costs  above  those  for  courses  where  tests 
are  centrally  scored  by  machine. 


V.  EVALUATION  OF  FEATURES  OF  THE  SOJET 
PROGRAM  DELIVERY  PROCEDURES 


A.  ACCEPTANCE  OF  REQUIREMENT  FOR  A  REGISTERED  SUPERVISOR 

When  a  student  enrolls  in  a  SOJET  course,  a  supervisor  must 
register  along  with  him  as  a  course  supervisor.  This  course  super¬ 
visor  is  expected  to  help  the  student  identify  his  training  require¬ 
ments,  take  responsibility  for  storing  and  maintaining  security  of 
test  material,  administer  and  score  subcourse  tests,  provide  immediate 
test  feedback  to  the  student,  and  assure  that  test  results  are  for¬ 
warded  to  the  Institute  for  Professional  Development. 

Survey  and  interview  data  relating  to  the  registered  supervisor 
requirement  are  summarized  in  Table  19.  A  majority  of  students  (60%) 
accepted  the  concept  of  having  a  course  supervisor  who  is  involved 
with  on-the-job  extension  training.  Moreover,  potential  students 
(89%)  and  supervisors  (98%)  expressed  a  willingness  to  accept  the  con¬ 
cept  of  course  supervisor.  Registered  supervisors  never  were  specifi¬ 
cally  queried  about  their  willingness  to  register  as  a  course  instructor. 
They  did  have  an  opportunity  to  comment  on  this  requirement  and  no 
complaints  were  received  about  it. 

One  of  the  potential  disadvantages  of  the  SOJET  program  is  that 
the  administration  of  a  test  requires  the  availability  of  two  persons, 
a  student  and  a  course  supervisor.  In  reply  to  a  query  about  this 
potential  problem,  85%  of  active  Arn\y  students  reported  that  their 
supervisor  usually  or  always  was  available:  82%  of  the  students  in 
National  Guard  and  Reserve  units  provided  a  similar  report  (Table  20). 
However,  28%  and  22%  of  active  Amy  and  NG/USAR  supervisors,  respec¬ 
tively,  reported  that  it  usually  or  always  was  difficult  to  find  a 
time  when  both  student  and  supervisor  were  available  for  testing  or 
for  review  of  test  scores  (Table  21,  Question  6 j ) . 

The  SOJET  program  is  based  in  part  on  the  assumption  that 
training  benefits  from  the  program  will  be  increased  by  involving 
first-line  supervisors  in  the  training.  Most  supervisor  and  student 
respondents  favored  the  active  participation  of  a  registered  course 
supervisor.  However,  as  noted  already,  the  requirement  to  have  a 
registered  supervisor:  (1)  Increases  administrative  costs  (p.  38  ) 
and  may  make  continued  student  participation  difficult  due  to  per¬ 
sonnel  (Supervisor)  turbulence  (p.  33  )•  If  so,  this  rests  on  the 
requirement  for  a  supervisor  to  administer  and  score  tests  as  opposed 
to  the  requirement  to  register  as  a  course  supervisor.  The  presence 
of  a  registered  supervisor  may  have  increased  the  extent  to  which  stu¬ 
dents  participated  in  the  SOJET  program,  but  there  are  no  data  to  sub¬ 
stantiate  this. 
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TABLE  19 


SUMMARY  OF  INTERVIEW  AND  SURVEY  DATA 
RELATED  TO  ACCEPTANCE  OF  THE  ROLE 
OF  COURSE  SUPERVISOR 


o 


98% 


SOURCE 
(ANNEX  A) 


of  potential  supervisors  were  willing  to 
register  as  a  course  supervisor 


Table  B-1G 


o  88%  of  potential  supervisors  were  willing  to 
administer  and  score  pre-and  posttests 


Table  B-9 


o  89%  of  potential  students  were  willing  to 
enroll  along  with  a  course  supervisor 


Table  B-U 


o  72%  of  persons  requesting  information  about  the 
program  reacted  favorably  to  a  description 
of  the  program 


Table  D-3 


o  60%  of  enrolled  students  expressed  a  preference 
for  having  a  course  supervisor;  another 
18%  said  they  "did  not  care  one  way  or  the 
other" 


Table  E-l 


o 


60% 


of  the  persons  interviewed  during  field 
trips  preferred  supervisor  involvement 
with  the  SOJET  program 


Table  B-12 
Table  B-16 
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TABLE  20 

STUDENT  REPLIES  TO  QUESTIONS  RELATED  TO  TESTING  PROCEDURES3 


Question  Number 

Component 

N 

Response  Options 

and  Question 

Never 

Ess msm 

HHZ1U1 

KIWT/1 

10.  Does  your  super¬ 
visor  administer 

AA 

17 

- 

- 

- 

100* 

- 

and  score  your 
pretests? 

NG/USAR 

60 

8* 

5* 

13* 

70* 

3* 

11.  Does  your  super¬ 
visor  administer 

AA 

17 

- 

- 

- 

94* 

6* 

and  score  your 
posttests? 

NG/USAR 

65 

5% 

3% 

7* 

82* 

2* 

12.  Is  your  super¬ 
visor  readily 

AA 

20 

- 

10* 

10* 

75* 

5* 

available  to 
administer  and 
score  your  tests? 

NG/USAR 

65 

51 

12* 

37* 

45* 

2* 

13.  Does  your  super¬ 
visor  provide 

AA 

17 

- 

- 

12* 

88* 

- 

feedback  to  you 
within  one  or  two 
days  after  you 
take  a  test? 

NG/USAR 

60 

5X 

13* 

r". 

CM 

52* 

2* 

14.  Do  you  get  to 
look  at  your 

AA 

16 

69* 

13* 

6* 

6* 

6* 

lesson  material 
when  taking  a 
pretest? 

NG/USAR 

58 

53* 

21* 

12* 

9* 

5* 

15.  Do  you  get  to 
use  your  lesson 

AA 

17  I 

65* 

12* 

12* 

- 

12* 

material  when 
taking  a  post¬ 
test? 

NG/USAR 

58 

57% 

14* 

17* 

9* 

3* 

aThird  Survey  Data 


TABLE  21 

SUPERVISOR  REPLIES  TO  QUESTIONS  RELATED  TO  TESTING  PROCEDURES* 


Question  Number 

Response  Options 

and  Question 

Component 

N 

Never 

ma 

Always 

6a.  Do  you  personally 
administer  and 

AA  ' 

17 

4% 

4% 

4% 

87% 

score  pretest? 

NG/USAR 

41 

2 % 

6% 

14% 

78% 

6b.  Do  you  personally 
administer  and 

AA 

17 

- 

- 

4% 

96% 

score  all  post- 
tests? 

NG/USAR 

37 

• 

6% 

4% 

89% 

6c.  Are  the  test 
scoring  guides 

AA 

17 

- 

22% 

48% 

30% 

easy  to  use? 

NG/USAR 

41 

6% 

10% 

45% 

39% 

6d.  Are  the  test 
scoring  guides 

AA 

17 

- 

22% 

43% 

35% 

accurate? 

NG/USAR 

39 

- 

10% 

56X 

33% 

6e.  Do  you  study 
lesson  or  test 

AA 

17 

26% 

35% 

26% 

13% 

material  before 
giving  a  test? 

NG/USAR 

41 

10% 

50% 

20% 

20% 

6f.  Do  you  allow 
students  to  take 

AA 

13 

48% 

38% 

5% 

9% 

tests  as  “open 
book"  exams? 

NG/USAR 

41 

55% 

18% 

16% 

8% 

6g.  Do  you  provide 
feedback  to 

AA 

15 

- 

5% 

18% 

77% 

students  within 

1-2  days  after 
tests? 

NG/USAR 

39 

2% 

4% 

23% 

71% 

6h.  Do  you  have  any 
problems  finding 

AA  | 

19 

88% 

8% 

- 

4% 

a  place  to  store 
tests? 

NG/USAR  | 

39 

75% 

19% 

4% 

2% 

61.  Do  you  have  any 
problems  main¬ 

AA  | 

15 

82% 

18% 

- 

• 

taining  test 
security? 

NG/USAR  | 

45 

96% 

4% 

6j.  Is  Is  difficult 
to  find  a  time 

AA 

15 

18% 

54% 

23% 

5% 

when  both  you 
and  the  student 
are  free  to  take 
and  to  review  a 
test? 

NG/USAR  | 

45  | 

H 

59% 

14% 

8% 

•Third  Survey  D«t« 
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A  SOJET-like  program  must  have  some  means  of  sending  test 
material  to  a  responsible  person  in  the  field.  In  the  SOJET  program 
this  is  accomplished  by  sending  the  material  to  a  registered  super¬ 
visor.  A  decision  to  continue  this  requirement  or  to  consider  other 
options  must  be  based  on  policy. 

Although  responding  favorably  to  the  concept  of  a  registered 
course  supervisor,  a  number  of  students  and  supervisors  expressed 
reservations  about  some  of  the  specific  activities  and  procedures 
required  of  course  supervisors.  These  requirements  are  reviewed  on 
the  following  pages. 


B.  ACCEPTANCE  OF  ENROLLMENT  PROCEDURES 

When  the  SOJET  program  first  was  implemented,  three  enrollment/ 
registration  forms,  and  at  least  one  Student  Training  Plan  had  to  be 
completed— a  process  which,  on  the  average,  required  34  minutes  of 
supervisor  time  (Table  22).  A  number  of  survey  respondents  complained 
about  the  time-consuming  and  complicated  nature  of  the  enrollment/ 
registration  process  (Table  23).  Also,  IPD  personnel  reported  that 
many  enrollment  applications  had  to  be  returned  because  the  Student 
Training  Plan  or  one  or  more  of  the  forms  were  filled  out  incorrectly. 
In  addition,  analysis  of  the  data  recorded  on  Student  Training  Plans 
led  to  the  conclusion  that  these  plans  were  not  being  filled  out  as 
intended  (see  pp. 57-61).  For  these  reasons  it  was  concluded  that 
enrollment  procedures  were  inadequate  and  unacceptable,  and  in  need  of 
immediate  revision. 

Currently  the  SOJET  program  uses  a  revised  set  of  enrollment  pro¬ 
cedures  and  forms.  The  number  of  enrollment  forms  was  reduced  from 
three  to  two  and  the  requirement  to  fill  out  and  return  a  Student 
Training  Plan  was  eliminated.  Also,  enrollment  instructions  were  re¬ 
written  to  make  them  easier  to  follow.  As  a  result  of  these  actions 
the  number  of  enrollment  applications  rejected  in  recent  months  has 
decreased  substantially. 


C.  ACCEPTANCE  AND  ADHERENCE  TO  TESTING  PROCEDURES 

The  SOJET  program  employs  a  variety  of  testing  features  which,  as 
a  total  package,  are  quite  unique.  Data  related  to  the  acceptance  of 
and  adherence  to  these  features  are  reviewed  on  the  following  pages. 

Storage  and  Security  of  Tests 

In  the  SOJET  program  all  lesson  materials  requested  by  the  stu¬ 
dent  are  mailed  to  him  immiJJTately  following  enrollment.  This  is 
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TABLE  22 


AVERAGE  TIME  TO  CONDUCT  COURSE  SUPERVISOR  ACTIVITIES 

(MINUTES)3 


ACTIVITY 

ACTIVE  ARMY 
SUPERVISORS 

Officers  Enlisted 

NG/USAR 

SUPERVISORS 

Officers  Enlisted 

Enroll  one  student 

34 

48 

29 

26 

Give  one  test 

30 

28 

39 

19 

Score  one  test 

17 

10 

16 

13 

Review  results  of  one 
test  with  student 

18 

22 

19 

31 

Send  results  of  one 
test  to  I PD 

13 

6 

11 

9 

Total 

112 

114 

114 

98 

3Th1rd  Survey  Data,  Question  2,  Supervisor  Questionnaire 


TABLE  23 


SUMMARY  OF  CRITICAL  COMMENTS  MADE  BY 
ENROLLED  STUDENTS  AND  REGISTERED  SUPERVISORS3 


Comments 

1st  Survey 

(NG/lfSAR  Respondents! 

3rd  Survey 

(AA  &  NG/USAR  Respondents) 

Supervisor 

(N-32) 

Student 

(N-38) 

Supervisor 

(N-86) 

Student 

(N-85) 

Enrollment  procedures  too 
complicated  and  time 

consuming 

6 

5 

8 

- 

Scorl ng/adml nl st rat 1 ve 
procedures  too  time 

consuming 

4 

- 

4 

- 

Errors  in  subcourse  material, 
tests,  and  scoring  guides 

1 

2 

11 

16 

Material  poorly  written/ 
material  and/or  test 

questions  confusing 

- 

- 

4 

4 

Way  needed  to  Identify  whl 
material  goes  with  whl cl 
student 

ch 

3 

i 

Master  list  needed  for 

material  sent  to  student 

and  supervisor;  cannot 

tell  If  anything  Is  missing 

2 

2 

• 

- 

Takes  too  long  to  get  material 
after  enrolling  s 

4 

More  time  needed  to  complete 
stucfy  | 

m 

m 

5 

Material  not  always  relevant 

m 

2 

4 

Supervisor  Is  not  needed 

- 

1 

3 

Need  “800“  hotline  to  I PD 

- 

- 

3 

Miscellaneous 

- 

14 

6 

aNumber  of  persons  who  made  same  or  similar  comment.  Replies  of 
active  Arny  and  NG/USAR  students  and  supervisors  have  been  combined. 
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called  “one  shot"  mailing.  Similarly,  all  corresponding  test 
materials  are  mailed  to  the  registered  course  supervisor,  who  Is 
responsible  for  storing  and  maintaining  security  of  the  test  material. 

When  the  SOJET  program  began  there  was  some  concern  as  to  whether 
supervisors  had  access  to  adequate  storage  facilities.  Interviews 
with  actual  and  potential  supervisors  and  students  Indicated  that  all 
supervisory  personnel  had  access  to  a  secure  storage  area.  During  the 
third  survey  81%  of  course  supervisors  reported  never  having 
storage  problems.  An  additional  14%  reported  that  they  sometimes 
have  such  problems  (Table  24).  Reports  of  storage  problems  were  most 
apt  to  come  from  National  Guard  or  Reserve  personnel.  This  reflects 
the  finding  that  some  National  Guard  personnel  became  the  course 
supervisor  for  as  many  as  12-15  students. 

Eighty-nine  percent  of  the  registered  supervisors  who  responded 
to. the  third  survey  reported  having  no  problems  maintaining  test 
security.  Security  problems  were  more  apt  to  be  reported  by  super¬ 
visors  In  the  active  Am\y  (18%). 

The  foregoing  findings  Indicate  that  the  "one-shot"  mailing  of 
lesson  and  test  material  Is  an  acceptable  practice— It  causes  few  If 
any  storage  and  security  problems. 

Complaints  not  related  to  storage  or  security  were  registered 
about  the  "one-shot"  mailing  approach  (Table  23).  When  a  person 
registers  as  a  supervisor  for  two  or  more  persons,  he  receives  test 
packets  for  each  student.  When  the  program  was  first  Implemented, 
there  was  no  easy  way  for  the  supervisor  to  match  test  material  with  a 
student.  In  response  to  complaints  about  this  problem  the  procedure 
was  changed.  Currently,  the  address  used  to  mall  out  test  material  to 
the  supervisor  also  lists  the  name  of  the  student  to  whom  the  material 
applies. 


Tests  Administered  and  Scored  by  Supervisor 

The  SOJET  program  course  supervisor  Is  responsible  for  adminis¬ 
tering  and  scoring  both  pretests  and  posttests.  Detailed  scoring 
guides  are  provided  for  each  test.  The  first  survey,  dealing  with 
National  Guard  and  Reserve  supervisors,  Indicated  that  test  adminis¬ 
tration  and  scoring  procedures  were  being  followed  rather  closely. 

The  findings  of  the  third  survey  for  enrolled  students  and  registered 
supervisors  are  presented  In  Tables  20  and  21. 

Both  enrolled  students  (Table  20,  Questions  10  and  11)  and 
registered  supervisors  (Table  21),  Questions  6a  and  6b)  reported  that 
most  course  supervisors  do  personally  administer  both  pre  and  post- 


TABLE  24 


SUMMARY  OF  SUPERVISOR  RESPONSES  TO  QUESTIONS  RELATING 
TO  SUBCOURSE  TEST  STORAGE  AND  SECURITY3 


Response  Options 

Question 

Component  - 

N 

Never 

Knag 

EMUmiCDOHi 

6a.  Do  you  have  any 
problems  finding 

AA 

24 

88% 

8% 

8% 

a  place  to  store 
tests? 

NG/USAR 

48 

75% 

19% 

4%  2% 

6b.  Do  you  have  any 
problems  main¬ 

AA 

22 

82% 

18% 

- 

taining  test 
security? 

NG/USAR 

51 

96% 

4% 

3Third  Survey  Data 


TABLE  25 

NUMBER  AND  PERCENT  OF  STUDENTS  FOR  WHOM  VARIOUS 
COMBINATIONS  OF  PASSING  AND  FAILING  PRETESTS  AND 
POSTTESTS  WERE  SUBMITTED 


Passing  or  Failing 

Status  of  Tests 

Submitted 

Pretest 
(N  -  273)® 

Test  Form 

Posttest 
(N  ■  273)* 

Passing  tests  only 

105  (39%) 

176  (64%) 

Mix  of  passing  and 
falling  tests 

84  (31%) 

14  (  4%) 

Falling  tests  only 

20  (  7%) 

— 

Did  not  submit  this 
test  form 

64  (23%) 

83%  (31%) 

3Number  of  persons  for  whom  either  or  both  a  pretest  and  posttest  was  submitted. 


49 


tests.  However,  supervisors  of  National  Guard  and  Reserve  units  were 
less  apt  to  be  personally  involved  in  test  administration  (82%  for 
posttests)  than  were  active  Amy  supervisors  (94%  Involvement).  The 
first  survey  of  NG/USAR  supervisors  revealed  that  the  training  NCO  of 
those  units  sometimes  administered  the  tests;  also,  on  occasion,  stu¬ 
dents  were  allowed  to  take  the  tests  at  home,  although  scoring  was 
always  done  by  a  supervisor  or  training  NCO.  These  findings  indicate 
that  course  supervisors  did  accept  responsbility  for  the  administra¬ 
tion  and  scoring  of  tests.  „ 


Test  Scoring  Guides 

The  test  scoring  guides  provided  to  course  supervisors  were  meant 
to  be  usable  by  non-subject  matter  experts.  Most  supervisors  (about 
80%)  did  report  that  these  guides  usually  were  easy  to  use.  They 
pointed  out,  however,  that  some  guides  contained  errors  (Table  21, 
Question  6d)  and  sometimes  were  difficult  to  use;  7%  of  the  NG/USAR 
supervisors  felt  that  the  guides  were  never  easy  to  use.  Most  of  the 
difficulties  with  the  scoring  guides  seem  attributable  to  the 
collation  errors  which  some  of  them  contained.  This  problem  is 
expected  to  be  removed  when  the  guides  are  revised  and  corrected. 
Generally,  the  data  indicated  that  the  use  of  test  scoring  guides  is 
an  acceptable  way  to  provide  test  scoring  guidance  to  non-subject 
matter  experts. 


Feedback  to  Students 


SOJET  program  course  supervisors  were  requested  to  provide  imme¬ 
diate  feedback  to  the  student,  an  action  that  cannot  be  accomplished 
using  the  traditional  delivery  sytstem  for  correspondence  courses. 
Almost  all  supervisors  complied  with  this  request.  Over  95%  of  the 
supervisors  reported  that  they  usually  or  always  provided  feedback  to 
students  within  one  or  two  days.  Active  Amy  students  confirmed  this 
(Table  20,  Question  13).  However,  almost  16%  of  the  NG/USAR  students 
reported  that  their  supervisor  seldom  or  never  provided  immediate 
feedback.  This  tended  to  be  confirmed  by  the  replies  of  their  super¬ 
visors  as  shown  in  Table  21,  Question  6g.  These  findings  can  be 
accounted  for  by  the  difference  in  availability  between  active  Amy 
and  National  Guard/Reserve  supervisors.  In  the  active  Amy  the  super¬ 
visor  and  student  may  be  in  daily  contact,  while  in  National  Guard 
and  Reserve  units  this  contact  may  be  on  a  monthly  basis.  Indeed,  some 
NG/USAR  students  commented  that  test  feedback  usually  was  given  them 
during  the  next  scheduled  duty  period. 


Use  of  Subcourse  Material 


It  seems  reasonable  to  presume  that  many  tasks  taught  In  the 
SOJET  courses  will  be  performed  on  the  job  using  appropriate  subcourse 
lesson  material  as  job  aids;  If  this  presumption  Is  correct,  the 
ability  to  pass  a  subcourse  test  using  SOJET  subcourse  lesson  material 
probably  Is  an  accurate  Indication  that  job  requirements  can  be  met. 

As  shown  In  Table  20,  25%  of  active  Arn\y  students  and  42%  of 
NG/USAR  students  reported  that  they  were  allowed  to  take  the  pretests 
as  an  "open  book"  test,  at  least  on  occasion.  Approximately  the  same 
percentage  of  students  said  they  took  posttests  as  "open  book"  tests. 
As  shown  In  Table  21,  an  even  higher  percentage  of  supervisors 
reported  the  practice  of  "open  book"  testing  (Question  6f). 

The  foregoing  findings  may  account,  In  part,  for  the  high  percen¬ 
tage  of  persons  who  passed  pretests  or  who  passed  posttests  on  the 
first  attempt.  The  SOJET  subcourse  material  Is  explicit  enough  so 
that  there  Is  little  reason  for  persons  to  fall  on  an  "open  book" 
test.  Furthermore,  It  should  be  noted  that  "open  book"  testing  1$  not 
specifically  prohibited  In  the  SOJET  program,  nor  In  most  other 
correspondence  courses,  for  that  matter.  It  should  be  remembered  also 
that  many  students  had  been  assigned  to  operation  cr  Intelligence 
positions  for  many  months  (Table  7,  page  24).  If  a  decision  Is  made 
that  students  should  not  consult  lesson  material  during  testing,  a 
specific  statement  to  that  effect  should  be  prominently  located  In 
each  SOJET  subcourse  test  scoring  guide. 


Test  Reporting  Requirements 

As  of  late  September  1979,  273  of  the  634  students  enrolled  as  of 
1  September  1979  had  submitted  tests  for  one  or  more  subcourses. 

Table  25  shows  the  number  and  percent  of  students  for  whom  passing  and 
falling  pretests  and  posttests  were  submitted.  The  data  Indicate 
that  the  majority  of  students  (77%)  who  submitted  any  kind  of  test 
submitted  a  pretest.  A  further  analysis  of  the  data  contained  In 
Table  25  revealed  that  61%  of  the  students  submitted  passing  tests 
only;  18%  submitted  only  passing  pretests;  231  submitted  only  passing 
posttests,  and  20%  submitted  a  mix  of  passing  pretests  and  posttests 
(Table  26). 

During  this  study  It  was  not  possible  to  estimate  the  percent  of 
persons  who  did  not  submit  a  falling  pretest.  At  a  minimum,  64  stu¬ 
dents  never  submitted  a  pretest;  they  submitted  only  posttests,  almost 
all  of  which  were  scored  as  "passing".  It  seems  probable  that  some  If 
not  most  of  these  students  failed  the  pretest  on  one  or  more  subcour¬ 
ses  but  never  reported  these  failures.  It  was  concluded,  therefore, 
that  the  requirement  to  report  falling  pretest  scores  was  to  some 
degree  unacceptable. 


TABLE  26 


PERCENT  OF  STUDENTS  FOR  WHOM  VARIOUS 
COMBINATIONS  OF  PASSING  AND  FAILING  PRETESTS 
AND  POSTTESTS  WERE  SUBMITTED 


Results  of  Tests  Submitted 

Form  of  Test 
Submitted 

All  Passing 
Tests 

Mix  of  Passing 
and 

Falling  Tests 

All  Fall  Inc 
Tests 

Total 

All  Pretests 

18% 

9% 

4% 

31% 

All  Posttests 

23% 

- 

- 

23% 

Mix  of  Pre- 
and  Posttests 

20% 

26% 

- 

46% 

Total 

61% 

35% 

4% 

Both  the  field  trips  to  CONUS  units  and  the  Initial  survey  of 
NG/USAR  students  and  supervisors  produced  Information  suggesting  that 
the  test  recording  procedures  were  In  need  of  change.  The  original 
procedures  reportedly  were  too  cumbersome;  the  requirement  to  submit 
falling  pretest  scores  was  not  being  followed  In  some  Instances;  and 
IPD  procedures  for  processing  SOJET  subcourse  test  results  were  time 
consuming.  To  correct  these  deficiencies  a  new  set  of  scoring  proce¬ 
dures  has  been  developed  and  Implemented  on  a  test  basis.  The  details 
of  these  procedures  are  described  In  the  SOJET  Program  Implementation 
Handbook,  Annex  B  to  this  report. 

In  general,  the  new  procedures  make  use  of  the  optical  scan  test 
recording  sheet  currently  employed  with  other  correspondence  courses. 
The  requirement  to  submit  pretest  scores  separately  has  been  dropped. 
Instead,  the  SOJET  course  supervisor  now  records  pre  and  posttest  re¬ 
sults  on  a  single  optical  scan  sheet.  When  the  student  has  passed  all 
subcourse  requirements,  this  scan  sheet  is  forwarded  to  IPD  for 
further  processing. 

As  a  result  of  these  new  procedures,  more  supervisors  should  be 
willing  to  report  pretest  results,  and  the  time  required  by  IPD  per¬ 
sonnel  to  process  the  test  results  will  be  reduced  to  that  required 
for  the  typical  correspondence  course. 


D.  REDUCTION  IN  OJT  TIME  DEMANDS  ON  SUPERVISOR 


Time  Requirements  of  Course  Supervisors 

The  SOJET  program  delivery  system  is  based  on  the  assumption  that 
time  spent  by  a  course  supervisor  to  supervise  students  will  be  more 
than  offset  by  a  reduction  in  the  time  they  otherwise  would  have  spent 
conducting  on-the-job  training.  The  validity  of  this  assumption  is 
difficult  to  demonstrate  because  very  few,  if  any,  NCOs  maintain  a 
record  of  the  time  they  expend  conducting  OJT. 

The  average  time  spent  by  supervisors  on  the  various  activities 
required  of  them  as  SOJET  course  supervisors  is  shown  in  Table  22. 

Time  required  to  enroll  one  student  is  about  35  minutes.  Thereafter, 
it  takes  about  75  minutes  to  administer  and  score  the  tests  for  one 
subcourse,  to  review  test  findings  with  the  student,  and  to  forward 
the  findings  to  IPD.  According  to  these  figures,  a  student  who 
enrolls  in  a  complete  operations  sergeant  course  consisting  of  19  sub¬ 
courses  would  consume  a  total  of  24  hours  of  course  supervisor  time, 
assuming  that  the  student  completed  his  program  of  instruction. 

Each  SOJET  subcourse  covers  a  specific  task,  some  of  which  are 
more  complicated  than  others  to  teach.  Assuming  that  these  tasks  were 
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learned  over  a  six-month  period  In  the  SOJET  program,  the  time  re¬ 
quired  per  week  by  a  course  supervisor  would  be  approximately  one 
hour.  This  probably  Is  a  lesser  amount  of  time  than  that  typically 
devoted  to  OJT.  Eighty-one  percent  of  NG/USAR  and  62%  of  active  Am \y 
supervisors  reported  that  SOJET  lesson  material  usually  or  always 
reduced  the  time  required  for  them  to  conduct  OJT  (see  discussion  of 
lesson  material,  and  Table  28). 

During  this  study  some' course  supervisors  reported  that  they  had 
or  anticipated  having  difficulty  finding  the  time  to  carry  out  their 
program  responsibilities.  The  effort  which  a  course  supervisor  must 
put  into  the  program  is  fairly  obvious,  while  the  benefits  to  be 
gained  from  the  program  seem  less  obvious  to  supervisors.  This  pro¬ 
bably  accounts  for  their  occasional  reluctance  to  participate  In  the 
program. 


Provision  of  High  Quality  Self-Study  Material 

One  goal  of  the  SOJET  program  was  to  develop  high  quality,  self- 
instructional  material.  This  research  did  not  attempt  to  directly  eva¬ 
luate  that  material,  but  the  various  surveys  did  solicit  information 
and  opinions  about  lesson  material  quality. 

Survey  data  (third  survey)  related  to  the  quality  and  relevance  of 
subcourse  material  are  summarized  in  Table  27.  Most  students  reported 
that: 

o  The  lesson  material  was  easy  to  understand. 

o  The  lesson  material  adequately  prepared  one  to  take  a 

posttest. 

o  The  lesson  material  usually  covered  tasks  performed  on  the 
job. 

o  The  procedures  described  in  the  lesson  material  usually 
were  similar  to  those  used  on  the  job. 

o  The  lesson  material  usually  or  always  helps  one  perform 
better  on  the  job. 

Course  supervisors  provided  similar  data,  as  shewn  in  Table  28. 

In  general,  supervisors  reported  that: 

o  The  SOJET  lessons  were  relevant  to  job  requirements. 

o  Student  job  proficiency  noticeably  improved  after 
students  studied  SOJET  lesson  material. 
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TABLE  27 


SUMMARY  OF  STUDENT  REPLIES  TO  QUESTIONS  ABOUT 
SOJET  LESSON  MATERIAL3 


Question  Number 

Component 

N 

Response  Options 

and  Question 

Never 

Sometimes 

iiRTEiira 

EU5EJQ9 

wmwim 

17.  Do  you  think  that 
SOJET  material  is 

AA 

!  — 

l5% 

60% 

25* 

- 

easy  to  under¬ 
stand? 

NG/USAR 

64 

2* 

9* 

58% 

29% 

2* 

18.  Does  the  study 
of  lesson 
material  ade- 

AA 

20 

- 

15* 

40% 

30% 

15* 

quately  prepare 
you  to  take  a 
posttest? 

NG/USAR 

63 

6* 

51% 

35% 

8* 

19.  Does  the  lesson 
material  cover 

AA 

20 

5* 

20% 

40% 

30% 

5* 

tasks  which  you 
perform  on  the 
job? 

NG/USAR 

64 

25* 

50* 

22% 

3* 

20.  Are  the  proce¬ 
dures  described 

AA 

20 

5* 

20* 

40% 

25* 

10* 

in  the  lesson 
material  similar 
to  those  you 
use  on  the  job? 

NG/USAR 

64 

22* 

59% 

16* 

3* 

21.  Does  study  of 

SCJET  material 

AA 

20 

- 

30* 

20% 

40* 

10* 

help  you  per¬ 
form  better  on 
the  job?  ■ 

NG/USAR  | 

64  | 

9* 

50% 

39* 

2* 

3Third  Survey  Data 
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TABLE  28 


SUMMARY  OF  SUPERVISOR  REPLIES  TO  QUESTIONS  RELATED 
TO  SUBCOURSE  EFFECTIVENESS  AND  JOB  RELEVANCE3 


Question 

Component 

N 

Response  Options 

No 

Sometimes 

i  Usually 

E3E2Z1 

mm 

12.  In  your  judgment 
are  SOJET  lessons 

AA 

25 

- 

8% 

60% 

32% 

- 

relevant  to  job 
requirements: 

NG/USAR 

57 

' 

7% 

54% 

32% 

7% 

13.  In  your  judgment 
does  the  job  pro- 

AA 

24 

4% 

13% 

54% 

21% 

8% 

ficiency  of  per¬ 
sonnel  noticeably 
improve  as  the 
result  of  study¬ 
ing  SOJET  lesson 
material? 

NG/USAR 

48 

15% 

27% 

50% 

8% 

14.  Does  the  use  of 
SOJET  lesson 

AA 

23 

13% 

22% 

35% 

26% 

4% 

material  reduce 
the  time  re¬ 
quired  to  train 
personnel  OJT? 

NG/USAR  | 

49  | 

2% 

16% 

16% 

61% 

4% 

aThird  Survey  Data 
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o  SOJET  lesson  material  reduced  the  time  required  to  conduct 
on-the-job  training. 

According  to  the  foregoing  replies,  both  students  and  supervisors 
were  of  the  opinion  that  SOJET  lesson  material  was  relevant  and  of 
high  quality.  However,  active  Army  students  and  supervisors  tended  to 
respond  less  positively  than  did  NG/USAR  personnel. 

It  should  be  noted  that  some  of  the  SOJET  lesson  material  did 
contain  errors  and  some  of  it  was  obsolete  (Table  23).  Persons  inter¬ 
viewed  in  the  field  and  respondence  to  all  three  surveys  noted  that 
this  was  the  case.  A  number  of  students  and  supervisors  reported  spe¬ 
cific  instances  of  error  or  obsolescence  to  IPD,  which  in  turn  forwarded 
this  information  to  the  appropriate  course  developers. 


Identification  of  Job  Requirements  and  Training  Needs 

A  supervisor  has  the  responsibility  for  identifying  job  require¬ 
ments,  for  identifying  those  specific  requirements  that  job  incumbents 
cannot  meet,  and  for  devising  means  to  correct  incumbent  deficiencies. 
As  an  aid  to  the  carrying  out  of  these  responsibilities,  the  potential 
SOJET  program  student  and  supervisor  were  provided  with  a  set  of  four 
Student  Training  Plans  (STP),  a  sample  of  which  is  shown  in  Figure  4. 

A  separate  STP  was  provided  for  each  of  the  four  basic  SOJET  courses. 

The  STP  for  a  particular  course  listed  the  subcourses  contained 
in  that  course  and  the  lessons  or  tasks  covered  by  each  subcourse.  In 
effect,  each  STP  provided  an  inventory  of  the  important  tasks  that  are 
performed  by  persons  in  a  particular  operations  or  intelligence  duty 
position  at  the  battalion  level.  The  course  supervisor  was  instructed 
to  review  all  the  STPs  and  to  select  the  STP  most  relevant  to  the 
potential  student.  The  supervisor  then  reviewed  the  tasks  listed  in 
the  STP  and  indicated  in  column  1  (see  Figure  4)  those  tasks  for  which 
the  potential  student  was  responsible.  In  column  2  the  supervisor 
indicated  the  degree  of  skill  required  to  perform  each  of  the  tasks 
checked  in  column  1.  Following  this,  the  supervisor  and  student 
reviewed  the  list  together  and  indicated  in  column  3  the  present  skill 
of  the  potential  student.  If  the  present  skill  level  for  any  par¬ 
ticular  task  was  less  than  that  required  for  the  job,  the  supervisor 
checked,  in  column  4  of  the  STP,  that  the  student  wished  to  order  that 
subcourse.  The  STP  was  returned  to  IPO  along  with  other  enrollment 
forms;  the  STP  served  to  verify  the  subcourses  enrolled  in  by  the 
student. 

Table  29  shows  the  average  estimated  skill  requirements  (column 
2)  and  the  average  estimated  current  skill  (column  3)  for  367  stu¬ 
dents.  The  data  have  been  subdivided  in  terms  of  the  skill  level  of 
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STUDENT  NAME: 


OPERATIONS  SERGEANT/ASSISTANT  OPERATIONS  SERGEANT  -  TASKS  FOR  AIL  BRANCHES  OF  THE  COMBAT  ARMS  _____ 

Col.  1  Col.  2  Col.  3  Col.  4 


Subcourai 

Number 

Subeouru/Uuon  Title 

Duty  Position:  OPS  SCT/ASST  OPS  SCT 
-  Combat  A  ms 

Tubs 

Rtquiftd 

(mtr 

V  or  # 

Skiff 

Re  quin  d 
(tatrn  M»§ 
SMsder  dm**/ 

Praktm 

sun 

ftmtrmt*/ 
aatakrn  ckttaa) 

Sakmam  hunt 

mm  f  ox  H 

OS 0001 

maintain  Pitanr  attack  __  _ 

wfl 

#1.  Use  Chargeout  Card. 

□ 

□ 

□ 

#2.  Determine  Publications  for  Library. 

□ 

□ 

□ 

#3.  Request  Publications  and  Update  Master  List. 

□ 

□ 

□ 

#4.  Supervise  Posting  of  Changes  to  Publications. 

□ 

□ 

□ 

0S0O02 

SUPERVISE  PREPARATION  OF  CORRESPONDENCE.*— , 

w-n 

#1.  Proof  Hllltary  Letters. 

□ 

□ 

□ 

#2.  Proof  Indorsements, 

□ 

□ 

□ 

#3.  Proof  Disposition  Forms. 

□ 

□ 

□ 

#4.  Proof  Non-Military  Letters  and  Memorandums. 

□ 

□ 

□ 

0S0003 

SUPERVISE  MAINTENANCE  OF  ARMY  FUNCTIONAL  FILES 

w  r~> 

OS 0004 

MANAHF.  TRAINING  RESniJRCES.. .  „  _ 

wp| 

11.  Coordinate  Training  Schedules. 

□ 

□ 

□ 

#2.  Prepare  Training  Ammo  Forecast. 

□ 

□ 

□ 

OSOOOS 

EVALUATE  CONDUCT  Or  TRAINING _  _ 

w  r-| 

#1.  Pr apart  for  Observation  of  Training. 

□ 

□ 

□ 

#2.  Observe  Conduct  of  Training. 

□ 

□ 

□ 

#3,  Evaluate  Training. 

□ 

□ 

□ 

OS 0006 

PREPARE  UNIT  READINESS  REPORT  __ 

wo 

” 

OS0007 

PLAN  AND  COORDINATE  CEREMONIES 

wJ~l 

#1.  Determine  Support  Requirements. 

□ 

□ 

□ 

12.  Determine  Sequence  of  Events, 

□ 

□ 

□ 

#3,  Prepare  Letter  sf  Instruction  (LOI). 

□ 

□ 

□ 

OSOOOS 

SUPERVISE  PREPARATION  OF  BR1EF1NC  CHARTS 

w  fn 

OS 0009 

PREPARP  l/UMKf!  <*ARfK  FOR  GROUND/ATR  MOVEMENT 

w  n 

OSOOIO 

BwpiBt  pop  npoirun  MnvrypvT  operations 

wn 

#1.  Organlte  the  Steps  to  be  Performed. 

□ 

□ 

□ 

12.  Develop  Vehicle  Load  and  Road  Movement  Plant. 

□ 

□ 

□ 

13.  Prepare  Road  Movement  Strip  Hap. 

□ 

□ 

□ 

14.  Assemble  Road  Movement  Operations  Ordsr 

□ 

□ 

□ 

(OPORD) . 

13.  Assemble  Information  for  FRAG  Order. 

□ 

□ 

□ 

osoou 

pprPAsr  roe  air  nrpiOYKENT 

n 

11.  Determine  Aviation  Requirements  to  Support 

□ 

□ 

□ 

Movement. 

12.  Prepare  Loading  Plans  for  an  Air  Movement. 

□ 

□ 

□ 

#3.  Assemble  Air  Movement  Operations  Order 

□ 

□ 

□ 

(OPORD). 

#4.  Prepare  Briefing  on  Movement. 

□ 

□ 

□ 

Figure  4.  Illustrative  Page  from  Student  Traininq  Plan  for  Operations 
Sergeant/Assistant  Operations  Sergeant  Course 


TABLE  29 


AVERAGE  RANKS  ASSIGNED  TO  SKILL  REQUIRED  AND  PRESENT  SKILL  LEVELS 
(COLUMNS  2  and  3)  OF  STUDENT  TRAINING  PLAN 


Student 

MOS  Skill 
Level 

Operations 

Sergeant 

Course 

Intelligence 
'  Sergeant 
Course 

Assistant/ 

Specialist 

Courses 

All 

Courses 

Combined 

liH 

Col  3U 

Col  2 

Col  3 

Col  2 

Col  3 

Col  2 

Col  3 

50 

3.65 

2.19 

3.53 

2.02 

3.67 

2.25 

3.59 

2.16 

40 

3.48 

1.85 

3.57 

2.12 

3.53 

1.53 

3.50 

1.86 

30 

3.08 

1.65 

3.58 

1.98 

3.58 

1.68 

3.32 

1.77 

20 

3.61 

1.79 

4.00 

1.00 

3.43 

1.43 

3.51 

1.5E 

10 

3.28 

1.13 

- 

- 

- 

- 

3.28 

1.13 

Average  for 
all  levels 

3.53 

1.19 

3.50 

2.06 

3.48 

1.53 

3.52 

1.97 

aCol  2  rankings  =  skill  required  of  job  holder 
bCol  3  rankings  *  current  skill  of  job  incumbent 
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the  student  and  in  terms  of  the  course  selected.  Those  persons  pro¬ 
cessed  under  the  Operations  Sergeant  course  include  those  who  enrolled 
only  for  the  basic  course  plus  those  who  enrolled  for  the  course  plus 
additional  subcourses.  The  same  is  true  for  STPs  processed  under  the 
Intelligence  Sergeant  and  the  Assistant/Specialist  courses.  Because 
of  a  low  N,  the  two  Assistant/Specialist  courses  were  combined.  The 
points  to  note  with  respect  to  Table  29  are: 

a.  For  all  courses  estimated  skill  requirements  (column  2) 
were  considerably  higher  than  current  skill  level 
(Column  3). 

b.  Persons  at  higher  grade  levels  tended  to  report  a  need  for 
higher  skill  requirements  and  also  tended  to  report  a 
higher  level  of  current  skill. 

The  foregoing  findings  are  what  would  be  expected.  Persons  at 
higher  grade  levels  usually  have  higher  skills  and  their  job  usually 
requires  them  to  possess  a  higher  skill  level,  in  part  because  they 
have  to  be  able  to  teach  a  variety  of  tasks  to  subordinates. 

The  discriminate  validity  of  the  STP  data  can  be  judged  in  part 
in  terms  of  the  pattern  of  responses  provided  by  the  students  and 
supervisors.  If  large  numbers  of  persons  assign  the  same  rank  to  all 
tasks  or  return  the  STPs  without  ranking  the  tasks,  that  is  an  indica¬ 
tion  that  the  respondents  are  not  carefully  discriminating  between 
task  importance,  skill  required  to  peform  the  task,  and  the  present 
skill  level  of  the  potential  student.  Table  30  shown  the  distribution 
of  ranks  provided  in  column  2  of  the  STP  (estimated  skill  required). 

In  completing  this  column  the  range  of  ranks  used  may  vary  from  1  rank 
(assign  all  tasks  a  rank  of  4,  for  example)  to  4  ranks.  As  shown  in 
Table  30,  41%  of  the  respondents  used  only  one  rank  when  estimating 
skill  requirements.  Further  analysis  showed  that  most  of  these 
persons  assigned  a  rank  of  4  to  all  tasks.  An  additional  31%  of  the 
respondents  used  two  adjacent  ranks  when  completing  column  2  (i.e.,  1 
and  2,  2  and  3,  etc.).  Most  of  these  respondents  used  a  rank  of  4 
with  the  occasional  use  of  rank  3. 

Students  and  supervisors  used  a  wider  range  of  ranks  when 
reporting  the  present  skill  level  of  students  (Table  30,  column  3). 
Only  24%  of  the  respondents  assigned  the  same  rank  to  all  tasks  while 
44%  used  a  range  of  either  3  or  4  ranks. 

The  foregoing  findings  suggest  that  a  substantial  minority  of 
supervisors  and  students  do  not  complete  student  training  plans  care¬ 
fully.  In  particular,  it  appears  that  many  raters  were  not 
differentially  rating  the  tasks.  Moreover,  approximately  one-half  of 
the  supervisors  either  left  column  2  of  the  STP  blank  or  used  only  a 
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TABLE  30 


RANGE  OF  RANKS  USED  TO  COMPLETE  SKILL  REQUIRED  AND  PRESENT  SKILL  LEVELS 
(COLUMNS  2  AND  3)  OF  STUDENT  TRAINING  PLAN 
(N  =  367) 


Student 

MOS  Skill 

Range  of  Ranks  Used  to 
Complete  Col  2a 

Range  of  Ranks  Used  to 
Complete  Col  3a 

Blank  1 

2 

3 

4 

Blank 

1 

2 

3 

4 

50 

14 

71 

56 

32 

10 

12 

30 

46 

55 

50 

40 

5 

37 

23 

12 

1 

4 

19 

21 

21 

13 

30 

7 

22 

18 

8 

2 

6 

20 

13 

13 

5 

20 

4 

17 

16 

6 

1 

4 

15 

9 

10 

6 

10 

- 

3 

1 

1 

- 

- 

2 

2 

1 

- 

Percent 

8% 

41* 

31% 

16% 

4% 

7% 

24% 

25% 

27% 

17% 

aNumber  of  persons  using  this  range  of  ranks 


TABLE  31 

STUDENT  AND  SUPERVISOR  PREFERENCE  FOR  MAILING  APPROACH 


Mailing  Options 

Component 

Student 

Preference 

Supervisor 

Preference 

One-shot  mailing 

AA 

75* 

78% 

NG/USAR 

76% 

91% 

Send  lessons  1  to 

AA 

25% 

19% 

3  at  a  time 

NG/USAR 

22% 

9% 

Other 

AA 

•» 

3% 

NG/USAR 

2% 

— 
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single  rank  when  estimating  skill  requirements.  Approximately  one- 
third  of  the  respondents  either  left  column  3  of  the  STP  blank  (7%)  or 
used  only  one  rank  when  completing  that  column.  On  the  survey 
questionnaires,  some  students  and  supervisors  did  report  that  the  STPs 
had  been  useful;  others,  however,  said  that  they  were  not  useful. 

.  A  Student  Training  Plan  has  many  potential  uses.  It  can  provide 
both  the  supervisor  and  the  student  with  an  overview  of  the  tasks  per¬ 
formed  by  operations  and  intelligence  personnel.  It  can  provide  a 
means  for  systematically  identifying  duty  position  requirements,  esti¬ 
mating  the  skill  level  required  for  performing  tasks  acceptably,  and 
assessing  the  capability  of  current  job  incumbents.  The  disadvantage 
of  an  STP  is  that  it  is  time  consuming  to  complete,  and  thus  the 
possibility  exists  that  persons  will  complete  it  in  a  perfunctory  and 
invalid  fashion.  In  view  of  the  uncertain  usefulness  of  the  Student 
Training  Plan,  and  the  need  to  reduce  the  time  required  to  enroll  in 
the  program,  it  was  decided  to  discontinue  the  requirement  to  submit  a 
STP  as  part  of  the  enrollment  application.  Instead,  the  STPs  now  are 
provided  strictly  as  an  aid  to  identifying  training  requirements. 

They  are  included  in  the  enrollment  package,  with  recommendations  on 
how  to  use  them,  but  supervisors  are  instructed  not  to  return  them  to 
I  PD. 


In  the  final  version  of  the  SOJET  program  it  is  suggested  that 
the  use  of  STPs  be  discontinued.  By  mid-1980  all  combat  arms  school 
catalogs  should  contain  a  listing  of  SOJET  subcourses.  Training  re¬ 
quirements  can  be  determined  by  a  review  of  these  listings.  Also, 
eliminating  STPs  will  reduce  the  printing  and  mailing  costs  associated 
with  the  SOJET  program. 


Training  Records  Centrally  Maintained 

For  most  OJT  programs,  training  records,  if  they  are  kept  at  all, 
are  maintained  at  the  unit  level.  The  SOJET  program  maintains  records 
at  a  central  location,  utilizing  specially  designed  procedures.  The 
goal  of  this  central  record  keeping  capability  is  to  relieve  super¬ 
visors  of  the  chore  of  maintaining  OJT  records.  Also,  central  records 
provide  a  means  for  students  to  receive  training  credits  and  promotion 
consideration. 

Field  acceptance  of  central  record  keeping  procedures  was  not 
directly  assessed  in  this  study.  However,  no  direct  complaints  were 
received  about  the  centralized  process.  As  noted  already,  complaints 
related  to  test  scoring  and  recording  procedures  were  received.  The 
nature  of  these  complaints  and  the  measures  devised  to  provide  solu¬ 
tions  to  them  were  discussed  under  test  administrative  and  scoring 
procedures. 


The  standard  procedures  developed  by  IPD  to  handle  student/ 
supervisor  queries  from  the  field  are  designed  to  provide  student 
representatives  at  IPD  with  easy  access  to  student  training  records* 
These  procedures  were  used  to  answer  questions  from  SOJET  students/ 
supervisors  and,  according  to  IPD  personnel,  worked  satisfactorily 
with  the  SOJET  program. 


E.  LOCAL  SEQUENCING  OF  TRAINING 


The  "One-Shot"  Mailing  Approach 


In  the  SOJET  program  all  subcourses  requested  by  a  student  are 
sent  to  him  immediately  following  enrollment.  Similarly,  all  cor¬ 
responding  test  material  is  sent  to  the  course  supervisor.  This 
procedure  is  called  "one-shot"  mailing.  The  one-shot  mailing  approach 
is  designed  to  allow  students  to  study  subcourses  in  an  order 
befitting  their  needs  and  interests  and  those  of  their  unit. 

During  the  field  visits  and  the  first  two  study  surveys,  con¬ 
siderable  support  was  expressed  for  the  one-shot  mailing  approach. 
Findings  related  to  this  procedure  and  obtained  during  the  third  sur¬ 
vey  are  contained  in  Table  31.  Both  students  and  course  supervisors 
expresssed  a  strong  preference  for  this  approach  over  that  used  with 
traditional  correspondence  courses.  A  somewhat  stronger  preference 
was  reported  by  National  Guard  and  Reserve  personnel. 

During  the  third  survey  students  and  supervisors  were  asked  to 
describe  how  they  decided  the  order  in  which  subcourses  were  studied. 
Replies  to  this  question  are  shown  in  Table  32.  Thirty-one  percent 
and  58%  of  active  Arn\y  and  NG/USAR  students,  respectively,  gave 
replies  which  suggested  that  they  did  take  advantage  of  the  scheduling 
opportunities  provided  by  the  one-shot  mailing  approach.  These 
replies  included:  studied  on  basis  of  unit  need,  order  determined  by 
supervisor,  order  based  on  student  interests,  and  first  studied 
material  least  knowledgeable  in.  The  remaining  students  said  either 
that  they  studied  subcourses  in  numerical  sequence  or  in  a  chance 
order,  or  that  they  studied  on  the  basis  of  factors  not  relevant  to 
student  or  unit  needs/interests.  As  seen  in  Table  32,  supervisors 
gave  a  similar  set  of  replies. 

The  extent  to  which  students  locally  sequenced  their  training 
should  be  reflected  in  the  order  in  which  subcourse  tests  are  sub¬ 
mitted  to  IPD.  This  order  can  be  determined  by  an  analysis  of 
student  training  records.  The  order  is  sometimes  distorted  because 
two  or  more  tests  may  be  submitted  concurrently,  with  no  indication  as 
to  which  one  was  taken  first. 
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TABLE  32 


MAJOR  FACTORS  USED  TO  DETERMINE  ORDER 
IN  WHICH  SUBCOURSES  WERE  STUDIED 


Factors 

Component 

Students 

(Na85) 

Supervisors 

(N=86) 

1.  Order  of  numbered  sequence 

AA 

2a 

3 

of  subcourses 

NG/USAR 

16 

16 

2.  On  basis  of  unit  need 

AA 

4 

11 

and/or  training  schedule 

NG/USAR 

11 

10 

3.  On  basis  of  student 

AA 

_ 

3 

needs  or  interests 

NG/USAR 

14 

16 

4.  Determined  by  supervisor 

AA 

1 

. 

NG/USAR 

6 

- 

5.  Studied  material  in  which 

AA 

2 

least  knowledgeable 

NG/USAR 

4 

5 

6.  Studied  easiest  material 

AA 

5 

2 

Or  shortest  lessons  first 

NG/USAR 

3 

2 

7.  Order  selected  at  random 

AA 

2 

w 

NG/USAR 

7 

4 

8.  Studied  in  order  received 

AA 

1 

in  mail 

NG/USAR 

3 

- 

9.  Miscellaneous 

AA 

«. 

NG/USAR 

- 

4 

10. No  response 

AA 

5 

8 

NG/USAR 

1 

• 

dNumber  of  persons  providing  same  or  similar  comments 
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An  analysis  of  c  .e  order  in  which  the  results  of  SOJET  subcourse 
tests  appeared  on  training  records  is  shown  in  Table  33.  The  students 
represented  in  this  Table  are  those  who  enrolled  for  at  least  a  complete 
course. 

The  medians  shown  in  Table  33  were  obtained  as  follows.  First, 
the  subcourse  number  of  the  first  test  submitted  was  determined. 
Secondly,  the  ordinal  position  of  each  of  these  subcourses  on  an 
appropriate  student  training  plan  was  determined.  For  example,  a  stu¬ 
dent  enrolled  in  an  Intelligence  Course  might  have  submitted  first  a 
test  on  a  subcourse  0S0015.  This  is  the  4th  listed  subcourse  on  the 
$TP  for  intelligence  sergeant  courses.  As  a  further  illustration, 
examination  of  five  students  enrolled  in  one  of  the  OPS  SGT  courses 
might  shown  that  they  first  submitted  tests  on  subcourses  OSOOOl, 

0S0003,  OSOOOl,  0S0005  and  0S0013.  Respectively,  these  are  the  1st, 

3rd,  1st,  5th  and  13th  listed  eubcourses  on  the  OPS  SGT  STP.  In  this 
illustration,  the  median  listed  order  of  first-submitted  tests  is  3. 
Using  similar  procedures  a  median  can  be  established  for  tests  sub¬ 
mitted  2nd,  3rd,  and  so  on. 

The  data  in  Table  33  indicate  that  there  is  a  mild  relationship 
between  the  order  in  which  a  test  is  submitted  and  the  order  in  which 
it  is  listed  on  a  student  training  plan.  This  relationship  is  fairly 
strong  for  OPS  SGT  courses  but  is  fairly  weak  for  INTEL  SGT  courses. 

In  general  the  data  support  the  survey  findings— at  least  33%  of  the 
students  tended  to  study  the  subcourses  in  an  order  which  reflected 
their  training  needs  and/or  the  needs  of  their  unit. 


Secondary  Advantages  and  Cost.  The  one-shot  mailing  approach 
provides  a  student  witn  a  complete  set  of  course  material  that  he  or 
others  in  his  unit  may  use  in  a  variety  of  ways.  During  the  third 
survey,  both  students  and  supervisors  were  asked  to  indicate  the 
various  ways  in  which  SOJET  material  was  utilized  in  their  unit.  The 
replies  are  summarized  in  Table  34. 

Fifty  percent  or  more  of  all  respondents  reported  that  the  lesson 
material  was  used  for  reference  purposes  and  as  a  aid  for  preparing 
for  SQTs.  Host  supervisors  reported  that  the  material  also  was  used 
to  train  persons  not  enrolled  in  the  program.  It  seems  important  to 
note  that  active  Army  supervisors  reported  more  extensive  utilization 
of  the  material  than  did  National  Guard  and  Reserve  supervisors;  pro¬ 
bably  this  is  because  many  National  Guard  and  Reserve  students  kept 
their  material  at  home.  Seventy-eight  percent  of  active  Army  super¬ 
visors  said  the  material  was  used  for  reference  purposes,  69%  report'4 
its  use  for  training  others,  and  56%  reported  its  use  at  an  aid  Tor 
preparing  for  SQTs.  Also,  35%  of  active  Army  supervisors  indicated 
that  the  material  was  used  to  inform  supervisors  about  how  to  perform 
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TABLE  34 

UTILIZATION  OF  SOJET  MATERIAL  AT  THE  UNIT  LEVEL* 


Ways  SOJET  Lesson 

Component 

Student 

Supervisor 

Material  Has  Been 

Replies 

Replies 

Useful 

As  reference 

AA 

82% 

80% 

material 

NG/USAR 

83% 

61% 

As  training 
material /job 

AA 

29% 

72% 

aids  for  persons 
not  enrolled  In 

NG/USAR 

64% 

52% 

program 

To  inform  super¬ 
visors  how  cer¬ 

AA 

29% 

40% 

tain  tasks  should 
be  performed 

NG/USAR 

42% 

35% 

As  an  aid  in  pre¬ 

AA 

59% 

60% 

paring  for 

NG/USAR 

54% 

50% 

Other 

AA 

6% 

8% 

NG/USAR 

12% 

6% 

*Th1rd  J>.  cudent  Ns  -  17  (AA)  and  59  (NG/USAR). 

Supervise.  (AA)  and  54  (NG/USAR). 
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certain  tasks.  It  appears  that  the  SOJET  material  has  been  used  in  a 
variety  of  beneficiJ  ways  by  persons  other  than  the  enrolled 
students.  In  part  this  is  a  side  benefit  of  the  one-shot  mailing 
approach. 

In  terms  of  costs  the  one-shot  mailing  approach  is  slightly  more 
costly  than  that  used  with  traditional  correspondence  courses.  In  the 
traditional  course,  three  lessons  are  first  sent  to  the  student; 
following  submission  of  at  least  one  test,  three  additional  subcourses 
are  mailed  out.  The  average  student  in  the  traditional  course 
receives  three  mailings  each  containing  three  subcourses.  In  the 
SOJET  program  all  subcourses  are  mailed  to  the  student  in  individual 
.packages,  and  the  average  student  receives  12  subcourses.  A  similar 
number  of  packages  containing  subcourse  test  material  must  be  sent  to 
SOJET  course  supervisors.  Because  of  these  differences  mailing  costs 
are  higher  for  the  SOJET  program  than  for  traditional  programs. 
However,  in  terms  of  mailing  costs  it  matters  little  whether  the  sub¬ 
course  test  packets  are  mailed  individually  or  in  packages  of  three, 
or  whether  they  are  sent  to  supervisors  or  included  with  student 
lesson  material.  Mailing  costs  are  related  primarily  to  the  weight  of 
the  material  being  mailed. 

The  benefits  received  from  the  one-shot  mailing  approach  are  dif¬ 
ficult  to  quantify  at  this  time.  At  least  33%  of  the  students  appear 
to  take  advantage  of  the  one-shot  mailing  approach,  since  they  do 
sequence  their  course  of  study  in  accordance  with  their  needs  and 
interests.  Also,  the  majority  of  students  and  supervisors  reported 
that  they  utilize  SOJET  material  in  a  variety  of  ways.  In  particular, 
active  Army  personnel  reported  that  the  material  is  extensively  used 
as  reference  material,  as  an  aid  to  training  others  in  the  unit,  and 
as  an  aid  to  preparing  for  SQTs.  For  certain  duty  positions  and 
tasks,  it  would  seem  important  to  provide  material  that  can  be  used 
for  informal  study  and  reference  purposes  even  though  it  never  is  a 
part  of  a  correspondence  course.  The  one-shot  mailing  approach  does 
serve  as  a  means  for  providing  such  material  at  the  unit  level. 
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VI.  FEEDBACK  TO  COURSE  DEVELOPERS  AND  PROGRAM  MANAGERS 


The  successful  management  of  correspondence  programs  requires  the 
collection  of  a  variety  of  data  that  can  be  used  by  program  managers 
to  administer  the  program  and  by  course  developers  to  make  appropriate 
revisions  to  lesson  material,  course  prerequisites,  and  so  on.  In 
this  section  some  of  the  reports  prepared  by  IPD  will  be  reviewed  and 
the  need  for  additional  reports  designed  especially  for  course  develo¬ 
pers  will  be  discussed  and  examples  provided. 


A.  RELEVANCE  OF  TRAINING  TO  LOCAL  REQUIREMENTS 

Currently  there  are  no  formal  procedures  for  collecting  data 
bearing  on  the  relevance  of  various  correspondence  courses  to  local 
job  requirements.  In  the  SOJET  program  the  Student  Training  Plan  was 
designed  so  that  the  supervisor  would  provide  information  about  the 
relevance  of  the  various  tasks  covered  in  one  or  more  of  the  SOJET 
courses.  As  already  noted,  complaints  were  received  about  the  time 
required  to  complete  the  Student  Training  Plan,  and  an  analysis  of  the 
data  provided  by  the  STP  suggested  that  much  of  the  data  might  be 
invalid.  For  these  reasons  the  formal  requirement  to  submit  a  Student 
Training  Plan  was  deleted  from  SOJET  procedures. 

Replies  obtained  during  the  third  survey  indicated  that  most  stu¬ 
dents  ar.d  supervisors  felt  that  the  SOJET  material  was  job  relevant 
(Table  28).  As  a  means  for  continuing  to  collect  such  information,  it 
is  suggested  that  at  the  end  of  each  SOJET  subcourse  lessen  packet  a 
form  be  provided  that  students  and  course  supervisors  can  use  to 
report  on  various  aspects  of  lesson  or  test  material.  Such  a  form  is 
employed  with  most  correspondence  courses.  The  typical  form  is 
designed  by  the  proponent  school,  usually  is  quite  general  in  nature, 
and  is  used  by  the  student  only  when  he  has  a  critical  comment  to  make 
about  the  lesson  or  test  material. 

It  is  suggested  that  such  a  form  would  be  more  useful  if  it  were 
slightly  structured.  Students  would  be  asked  to  return  the  form  when 
they  had  a  critical  comment  to  make  about  one  or  more  of  the  topics 
listed  on  the  form.  An  example  of  such  a  form  is  shown  in  Figure  5. 
Item  le  on  that  form  asked  the  student  to  indicate  whether  the 
material  covered  in  a  particular  subcourse  was  relevant  to  his  duty 
requirements.  If  the  student  returned  the  form  for  any  reason,  he 
would  answer  this  and  the  other  questions  listed  under  Item  1  of  the 
form. 


In  the  SOJET  program  there  needs  to  be  a  form  that  course  super¬ 
visors  can  use  to  comment  about  course  and  test  material.  Figure  6 
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Student  Subcourse  Critique  Form 


Subcourse  Number 


Instructions:  Return  this  form  if  you  have  critical  comments  to  make. 

a.  Record  number  of  subcourse  you  are  commenting  about. 

b.  In  Item  I  below  check  all  comments  that  apply  to  subcourse. 

c.  In  Item  2  below  explain  nature  of  comment  &  provide  solution 

if  you  have  one. 

d.  Use  additional  sheets  of  paper  for  comments  if  needed. 

e.  Fold  form  and  staple  twice  and  mail. 

Item  1 :  Check  (✓)  all  comments  that  apply  to  listed  subcourse 

a.  _  Lesson  material  incorrect  or  obsolete. 

b.  _  Lesson  material  difficult  to  understand. 

c.  _  Lesson  material  and  test  questions  do  not  agree. 

d.  _ Answer(s)  to  some  test  questions  are  incorrect. 

e.  _  Lesson  material  not  relevant  to  job  duties. 

f .  _ _ Other:  _ 

g.  Other: 


Item  2:  Provide  more  detailed  description  of  comment  and  suggest  a 
solution  if  you  have  one. 


Figure  5.  Illustration  of  Student  Subcourse  Critique  Form 
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Supervisor  Subcourse  Critique  Form 
Subcourse  No.  _ 

Instructions:  Return  this  form  if  you  have  critical  comments  to  make. 

a.  Record  number  of  subcourse  you  are  commenting  about. 

b.  In  Item  1  below  check  all  comments  that  apply  to  subcourse. 

c.  In  Item  2  below  explain  nature  of  comment  &  provide  solution 

if  you  have  one. 

d.  Use  additional  sheets  of  paper  for  comments  if  needed. 

e.  Fold  form  and  staple  twice  and  mail. 

Item  1:  Check  (/)  all  comments  that  apply  to  listed  subcourse 

a.  _ Test  scoring  guide  is  difficult  to  use. 

b.  _ Answers  provided  in  test  scoring  guide  are  incorrect. 

c.  _ Lesson  material  incorrect/obsolete. 

d.  _  Lesson  material  difficult  to  understand. 

e.  _  Lesson  material  not  relevant  to  job  duties. 

f .  _ Other: _ 

g.  _ Other:  _ 

Item  2:  Provide  a  more  detailed  description  of  comments  and  suggest  a 
solution  if  you  have  one. 


Figure  6.  Illustration  of  Supervisor  Subcourse  Critique  Form 


contains  an  illustration  of  a  Supervisor's  Subcourse  Critique  Form  for 
this  purpose.  It  is  suggested  that  this  form  be  located  at  the  end  of 
each  subcourse  pretest  scoring  guide.  As  illustrated,  the  form  would 
contain  a  short  list  of  questions  to  be  answered  by  a  check,  followed 
by  a  more  detailed  explanation  of  the  complaint  being  registered. 

An  indirect  way  of  determining  the  job  relevance  of  subcourses  is 
to  determine  the  rate  at  which  the  subcourses  are  requested.  IPD  pro¬ 
duces  a  monthly  report  showing  the  demand  rate  for  all  subcourses  for 
the  past  12  months.  By  scanning  this  report  it  becomes  obvious  which 
subcourses  are  in  high  and  in  low  demand.  Presumably  a  high  demand 
for  a  subcourse  would  be  an  index  of  a  perceived  relevance  at  least  of 
the  subcourse  title.  Table  36  shows  the  number  of  SOJET  subcourses 
which  were  issued  from  May  1978  through  April  1979.  Subcourses  0S0001 
and  OS0027  are  examples  of  high  and  low  demand  subcourses, 
respectively. 

Table  35  also  shows  the  percent  of  students  who  submitted  a  test, 
pre  or  posttest,  passing  or  failing,  for  each  SOJET  subcourse.  As  an 
illustration,  during  the  period  from  May  1978  through  April  1979,  340 
sets  of  subcourse  OSOOOl  lesson/test  material  were  issued.  During  the 
period  from  May  1978  through  June  1979  some  kind  of  test  on  OSOOOl 
material  had  been  received  from  27%  of  the  stuidents  to  whom  the 
material  had  been  issued.  This  information  can  be  used  by  course 
developers  to  identify  those  subcourses  for  which  tests  are  seldom 
submitted.  They  then  can  examine  such  subcourses  to  determine  if 
their  tests  are  too  difficult  or  if  their  content  is  obsolete  or  no 
longer  job  relevant. 

Figure  7  contains  an  illustration  of  the  proposed  report  for  pro¬ 
viding  subcourse  issue  information  and  subcourse  test  submission 
rates  to  course  developers.  This  form  shows  for  each  subcourse  the 
number  of  subcourses  issued  per  month,  the  number  of  passing  subcourse 
tests  received  per  month,  and  the  total  number  of  subcourses  issued 
and  tests  received  during  the  past  12  months.  This  report  is  somewhat 
similar  to  the  subcoursc  utilization  report  now  prepared  by  IPD. 
However,  additional  programming  would  be  required  to  develop  the  test 
submission  data. 


B.  EXTENT  TO  WHICH  TRAINING  REACHES  TARGET  AUDIENCE 

Currently  IPD  has  few  procedures  for  determining  the  degree  to 
which  training  readies  the  appropriate  audience.  One  report  shows  the 
type  of  student— defined  as  Active  Army,  National  Guard,  Navy, 
Civilian,  and  so  on— uhu  enrolls  in  various  programs.  The  absence  of 
more  detailed  student  profile  information  is  in  part  due  to  the  lack 
of  space  for  capturing  additional  data  on  DA  Form  145,  the  form  used 
to  enroll  in  a  correspondence  course. 
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TABLE  35 


NUMBER  OF  SUBCOURSES  ISSUED  THROUGH  APRIL  1979 
AND  PERCEMT  0F  TESTS  SUBMITTED  THROUGH  JUNE  1979 


Subcourse 

No. 

Issued 

Tests! 

I  Subcourse 
No. 

Issued 

Tests 

Subcourse 

No. 

Issued 

Tests 

340“ 

. rre" 1 

45 

7% 

2 

365 

23% 

24 

no 

6% 

66 

44 

14% 

3 

367 

22% 

25 

113 

11% 

67 

33 

9% 

4 

280 

27% 

26 

123 

8% 

68 

31 

10% 

5 

284 

19% 

27 

57 

7% 

69 

38 

5% 

6 

258 

16% 

28 

60 

5% 

70 

47 

19% 

7 

272 

17* 

29 

84 

12% 

0N0711 

8 

282 

18* 

30 

88 

6% 

12 

Recent 

Issue 

9 

275 

16% 

0A6001 

62 

47% 

13 

10 

286 

12* 

02 

117 

19% 

15 

11 

255 

US 

03 

59 

37% 

OR 1490 

69 

9% 

12 

282 

20% 

04 

112 

17% 

91 

46 

13% 

13 

282 

15% 

05 

112 

22% 

92 

46 

20% 

14 

278 

18% 

06 

111 

18% 

93 

45 

16% 

15 

130 

23% 

07 

53 

47% 

94 

49 

16% 

16 

131 

16% 

08 

50 

30% 

95 

78 

12% 

17 

126 

19% 

09 

53 

27% 

96 

83 

12% 

18 

128 

18% 

010860 

28 

0% 

97 

83 

7% 

19 

150 

14% 

61 

37 

0%  1 

98 

87 

11% 

20 

141 

24% 

62 

41 

o%  1 

21 

146 

20% 

63 

41 

0%  I 

22 

146 

18% 

64 

52 

4% 

i 

Percent  of  students  submitting  any  kind  of  test,  pre  or  posttest,  passing 
or  falling 
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In  addition  to  the  DA  145  Form  used  to  enroll  students,  the 
SOJET  program  used  a  special  form  to  register  course  supervisors.  The 
original  version  of  this  later  form  collected  a  variety  of  profile 
information  about  both  students  and  supervisors.  Some  of  the  super¬ 
visor  profile  information,  obtained  from  the  supervisor  registration 
form,  is  shown  in  Tables  36  and  37;  it  is  of  interest  but  is  not  cru¬ 
cial  to  the  successful  revision  of  course  material.  The  currently 
used  version  of  the  supervisor  registration  form  collects  very  little 
profile  information. 

The  amount  of  student  profile  information  that  can  be  collected 
on  future  SOJET  student  enrollment  forms  will  be  determined  primarily 
by  the  extent  to  which  additional  data  can  be  r  corded  on  a  revised 
version  of  DA  Form  145.  In  turn,  this  will  depend  on  the  deletion  of 
certain  data  from  the  present  form  (Figure  8).  Analysis  of  section  4 
of  DA  Form  145  has  shov/n  that  six  additional  columns  of  data  could  be 
captured  on  that  form.  In  addition,  data  pertinent  to  "RYE  Date-D^y" 
and  “EYE  Date-Dayu  could  be  deleted  because  this  information  is  not 
needed.  This  would  free  space  on  the  present  form  for  collecting 
other  types  of  data,  specifically  student  MOS  and  skill  level,  number 
of  months  in  current  duty  position,  and  total  number  of  months  in 
related  duty  positions.  For  more  information  about  revisions  to  this 
form  see  Annex  B. 

To  make  use  of  this  new  information,  programs  should  be  developed 
so  that  reports  could  be  prepared  to  display  student  profile  infor¬ 
mation  in  response  to  a  variety  of  questions.  These  questions  should 
include:  (a)  what  are  the  characteristics  of  students  enrolled  in  a 
particular  course;  (b)  what  are  the  characteristics  of  persons  who 
pass  or  fail  particular  subcourse  tests;  (c)  what  type  of  person 
enrolls  but  never  submits  a  test;  and  so  on.  In  response  to  these  and 
similar  questions,  a  report  similar  to  that  shown  in  Figure  9  would  be 
produced. 


C.  COST  EFFECTIVENESS  OF  INDIVIDUAL  SUBCOURSES 

Currently  IPD  produces  a  number  of  monthly  reports  that,  in  one 
way  or  another,  can  be  used  to  judge  the  cost  effectiveness  of  sub¬ 
courses.  One  report  shows  sub-course  demand  ra:e  over  a  12-month 
period.  It  is  used  primarily  to  determine  reorder  points  for  subcour¬ 
ses.  However,  It  also  can  be  used  to  de. ermine  those  subcourses  for 
which  there  is  little  demand— information  that  would  be  of  use  to 
course  developers. 

As  noted  previously,  a  type  of  report  that  would  be  useful  to 
course  developers  is  one  showing  the  ratio  of  tests  submitted  to  sub¬ 
courses  ordered.  Table  35  (page  73)  contains  this  type  of  information 
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PROGRAM  COURSE  SUPERV 
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TABLE  37 


DISTRIBUTION  OF  ACTIVE  ARMY  SUPERVISORS  BY  DUTY  POSITION 
AND  COURSE  REGISTERED  IN® 


Courses 

Supervisor 

Duty 

Position 

Operations 

Sergeant 

Intelligence 

Sergeant 

Operations 

Assistant 

Specialist 

Intelligence 

Assistant/ 

Specialist 

Totals 

Unit 

Commander 

1 

_ 

m 

m 

OPS  Offi¬ 
cer/Asst 

OPS 

Officer 

24 

1 

25 

OPS  SGT/ 
ASST  Ops 

Sgt 

17 

1 

5 

m 

23 

INTEL 

Officer/ 

Asst 

INTEL 

Officer 

2 

16 

4 

22 

INTEL 

Sgt/Asst 

INTEL 

SGT 

6 

3 

9 

Other 

22 

7 

- 

1 

30 

®If  a  supervisor  registered  In  both  a  senior  and  junior  course,  he  was 
assigned  to  the  senior  source. 
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STUDENT  ENROLLMENT  APPLICATION 
Supervised  On-The-Job  Extension  Training  (SOJET) 
ARMY  CORRESPONDENCE  COURSE  PROGRAM 

For  uoo  of  thlc  for*,  see  SCJET  Court#  Guide.  Th#  proponent  agency  ia  TRADOC. 


_  •  DATA  REQUIRED  AY  THE 


iflgjnwai 


AUTHORITY:  10  U1C  AO II  <•)  u>d  <0> 

PRINCIPAL  PURPOSE:  To  obtala  ialonuUM  mnwry  by  Army  ichoob  t#  administer  student  penicipeO#*  in  IN*  Army  correrpoadenoe 
count  program. 

ROUTINE  USES:  Used  by  Army  ichoob  to  obtain  baric  4ata  ntcdtd  t#  determine  eligibility  l#r  enrolment.  proeem  applications,  hmH 


ROUTINE  USES:  Uatd  by  Army  acboob  to  obtain  baric  data  needed  to  determine  eligibility  for  enrolment,  proeem  application#,  ■ 

lain  student  records,  and  perform  all  etbrt  admmirtntiee  functions  w  He  rent  in  etudent  administration. 

DISCLOSURE:  Mandatory.  Pi  Du  re  to  pronde  tHir  tnformaUon  could  result  in  the  applicant  not  being  able  to  ponleioate  in  the 


SUBMITQNECOTY.  US  IS'STXUCTIONS 


1*  THRU:(Unlt  t0  whlch  ••*i*n*d>  TITLE  OF  APPROVING  OFFICIAL 


WIT  ADDRESS  LIME  I  WIT  DESIGNATION  (Hoy  not  bo  Ult  blank) 


WIT  ADDRESS  LINE  2  f.  0.  SOX  OR  STREET  (Hoy  bo  loft  blonk) 


SEQ  TRANS 

NO  CODE 


WIT  ADDRESS  LINE  )  CITY.  POST  OR  APO 


IS  It  i I  31  34  31 J? 


2.  FROM:  (Hailing  oddroat  to  which  aubcourooo  oro  to  bo  aont) 

LAST  NAME  -  FIRST  HAW  -  MIDDLE  INITIAL 

SEQ  TRANS 

NO  CODE 


STUDENT  ADDRESS  LINE  I  WIT  DESIGNATION  OR  P.0.  SOX  OR  STREET  (Hoy  not  bo  loft  blonk 


SO  CODE 

a  m 


2.  t  REQUEST  EHROLLNDfT  IMt 


a.  Chock  boro  If  on ro linen t  lo  for  o  caaploto  cor 
_ Operations  SCT  Operation#  Aaat/lpoc 


(chock  only  ono  course  till#  and  on#  branch) 

__  Amor  _ field  Artillery 


_  !  Meingo*«o  SCT  Intolllionca  Aoot/Spoc  _  Infantry  _ Air  Del  onto  Artillery 

b.  _  Orach  boro  If  enrol loom  lo  for  oclactod  aubcourooo  only,  llot  those  aubcourooo  In  tten  1  of  tbo 

Suporvtooc'o  Ret la trot to#  for*. 

c.  ___  Otock  boro  If  you  or#  currently  or  hove  boon  prowtoualy  enrolled  to  o  ROJTT  otudoot. 


wniwnnmww  m  ihttitt mr: 


Mt  IMaOfD * 


SCHOOL  RECORD 


—5BE-  jsaf  . - assn lx* - -  a  K  rn 

i^s_±p  dp  iti  i  i  l  l  i  nj  e?  g  g  O 

COURSE  SUbCOUftSI  NUMERIC  W  C*auP  KUMSU 

NUMSf  R  lUbC  OUR  SC  NWBCR  *10  COOf  CRAM  OR  Jfi  _ 

&  ensiq  cep  gpp  g  \£n 

roi  ecu  tvu  i/c  ...  _ 

Hi  cow  owa ,  «*«  ...  me — ,  ,ttf  SS. 

cd  (=□  rm  r?a  eg 

M  SI  If  1 3  M4  it  it  it  *>  11  '* 

RYE  fUTT  w  CREDIT  HOURS  ACObRfUTt© 

*  Q  m  m  m  c*  fi| 

77  71  «•  If  dd  it  f*  tt  13  13  it  ;j  ti  is  or  *4 


rw*xiwi‘Z*uu.-i-mi+rm 


Sto  fhAN> 


SUKOUiSC  NtbOCI 


Subcouui 
sco  eooc 


NUMERIC 

CUM 


(NR  C«0UP  NUMbU 

VAR  OR  ID 


Id  3$ 


WT 

COOT  SRANC  RAW 

CS  CD  □=□ 


Si  m  it  tt 

EYE  Herr 

MY  month 


am  nn  porn  us.  july  im 


i;»  4l  RrsfrtCu.  t  l  V»|  -WliurtlrJi 


Fiqure  8.  SOJET  Program  Student  Enrollment  Application  Form 
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STUDENT  DATA: 
STUDENT  Sfll 


NAT't  CUARD/USAR 
RANK  (ZF  AfFLZCAILE) 

cm 


^  __  enrollment  date 

CARD  ACT IT! 

IT  NO.  DAT  Harm  TEAR  BUTT  RAW 

lftt  cs3  bi  g=d 


TOTAL  MQNTRS  Of 

Acrm  oun  (arht) 


25  27 


TRIMART  HOS 


32  34 


TOTAL  Hams  ZMi 

CURRENT  HOS  CURRENT  SKILL  LEVEL 

rm  rm 


TOTAL  HOmiS  Dl' 
PRESENT  PRESENT  DUTY 
Dtmr  POSITION  (CURRENT  UNIT) 

a  m 

<1  42  43 

TTPE  OP 

enrollment  i 


(1) 

Prteeat  Sutjr  .(}) 
9"l*i**  m 
(Check  Mt): 

(4) 


TOTAL  MONTHS  IN 
OPS /INTEL  DUTT 
(ALL  ASSIGNMENTS) 


45  47 

.  Opiricliu  SOT  (3) 

.  Aaat.  OftntlMM  SCT  (4) 

.  httlU|«Mt  SCT  (7) 

Allt.  lKllltiiMi  SCT 


COURSE  COURSE 

RRANCR  DUTY  POSITION 


41  49 

OMratlMi  Aiit/S^c 
lalelliftace  AMt/S^M 
Other  (pleeee  yrl»t)t 


*•  1  REALIZE  I  MUST  COMPLETE  THIS  INSTRUCTION  WITHIN  THE  TIME  LIMITS  ESTABLISHED  IT  THE  SCHOOL  COMMANDANT 
AND  1  INTEND  TO  XEET  THE  REQUIREMENTS 


mot  SkUt  LtmU 


Mr*#  Ana  •/  A**Um»n  t»P>  to  tofc; 


U(ir*m««  H'UUIri  II  MnllMM  ^mIIIUiiImu,  nMIiiIm  el  ifylUiiliM  ato  (mmi  available  ara  taauiaN  la 
Iba  SOJET  Cavraa  Cvtto. 


IMST MUCTIONS  TO  APPLICANTS 

CompItU  by  lafikfy  Work  priaLinf  only  in  areal  that  art  aat  aheSatL  If  WSMMail  apart  ia  rvfai—4.  ilUtk  aapawU  aHeel*. 
00  HOT  TUI  ia  iheOel  artaa.  Artaa/Worki  pH* eh  eaauia  baab  aiarka  may  W  w*4  la  ktypaack  4aU  tor  pat  ia  atlamatoS 
■ytitma;  aaiar  otWy  aaa  cterertet  par  hath  mark.  94.,  ^Q|7  |«|3l  1  | >  1 1  |4jTl  I >|>  |0j 


ITEM  1.  Oa  »»  Rm  baa  flat  NO#  ml  imatai  ailtotol;  to*  iitapi 
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Figure  8a.  SOJET  Program  Student  Enrollment  Appl ication(2nd  side) 
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for  SOJET  subcourses.  For  example,  for  subcourse  0A6007,  53  subcour¬ 
ses  were  Issued  during  the  previous  12  months  and  the  rate  of  test 
return  was  47%.  For  the  SOJET  program  this  is  a  very  high  rate  of 
test  return.  On  the  other  hand,  subcourse  0S0030  was  Issued  to  88 
persons  and  the  rate  of  test  return  was  only  6%. 

Of  course,  the  reasons  for  high  demand  for,  or  low  rate  of  test 
return  for,  any  particular  subcourse  would  have  to  be  determined  by  an 
analysis  of  written  and  interview  comments  provided  by  the  students. 

In  any  event,  the  data  in  Table  35  do  suggest  that  certain  subcourses 
are  cost  effective  while  other  are  not.  It  is  proposed  that  the 
actual  report  containing  the  foregoing  information  be  formatted  as 
shown  in  Figure  7  (page  74).  The  course  developer  would  have  to 
determine  the  percent  of  test  returns,  but  this  should  be  simple  to 
accomplish. 


D.  SUBCOURSE  QUALITY  CONTROL 

Two  types  of  feedback  information  are  of  special  Interest  to 
course  developers.  One  type  identifies  deficiencies  In  lesson  and 
test  material,  and  the  other  analyzes  test  results.  The  typical 
correspondence  course  uses  a  student  feedback  form  to  obtain  infor¬ 
mation  about  lesson  and  test  material  deficiencies.  This  information 
Is  forwarded  to  course  developers  by  IPD  personnel. 

As  already  noted  (Table  20),  most  complaints  about  the  SOJET 
program  concerned  the  mismatch  between  lesson  am"  test  material.  These 
complaints  seemed  due  to  editing  and  collation  deficiencies,  and 
they  should  be  corrected  by  appropriate  School  personnel  prior  to 
reprinting  of  the  material.  It  is  suggested  that  the  revised  sub¬ 
course  material  contain  a  student  subcourse  critique  form  as  shown  in 
Figure  5,  and  that  the  scoring  guide  for  each  subcourse  pretest  con¬ 
tain  a  supervisor  subcourse  critique  form  as  shown  in  Figure  6. 

IPO  employs  an  item  analysis  report  to  present  information  about 
passing  and  failing  rates  for  subcourse  test  items.  8ecause  of  dif¬ 
ferences  in  test  recording  and  reporting  procedures,  that  report 
could  not  be  prepared  for  SOJET  courses.  However,  as  described 
earlier,  the  SOJET  test  recording  procedures  have  been  modified.  Now 
the  standard  test  recording  form  used  for  other  correspondence  courses 
can  be  utilized  to  report  test  results  for  SOJET  subcourses.  Hence  it 
will  be  possible,  in  the  future,  to  produce  item  analysis  reports  for 
SOJET  courses.  Figure  10  shows  how  such  a  report  will  look.  In  the 
SOJET  program,  test  scoring  is  on  a  GO/NO  GO  basis.  Therefore,  the 
item  analysis  report  will  show  the  number  of  students  who  failed  test 
requirements  on  the  pretest,  the  first  posttest,  or  the  second  post¬ 
test.  Additional  information  about  the  revised  test  recording  prcce- 
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Subcourse  Edition  Lesson  or  Question  or  Correct  Percent 
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gure  10.  Illustration  of  Item  (Requirements)  Analysis  Report* For  A 
4-Requirement  SOJET  Subcourse  Test 


dures  is  contained  in  the  SOJET  Program  Implementation  Handbook, 
Annex  B. 


E.  AMOUNT  OF  PERSONNEL  TURBULENCE 

The  original  version  of  the  SOJET  program  supervisor's  guide  con¬ 
tained  a  form  to  be  used  to  notify  IPD  when  either  the  supervisor  or 
the  student  was  transferred,  or  for  some  reason  was  no  longer  able  to 
continue  in  the  course  worker  to  continue  In  the  role  of  a  super¬ 
visor.  The  information  obtained  on  that  form  could  have  been  used  to 
develop  data  bearing  on  personnel  turbulence.  However,  the  data  never 
were  used  in  this  manner.  It  Is  doubtful  whether  large  numbers  of 
students  were  even  aware  of  the  existence  of  this  form,  and  many 
supervisors  probably  forgot  about  It. 

It  is  suggested  that  a  revised  version  of  this  form,  as  shown  in 
Figure  11,  bo  used  to  collect  information  about  the  status  of  super¬ 
visors  and  students.  In  addition  to  alerting  IPD  to  the  need  to  help 
a  student  obtain  a  new  supervisor,  the  form  can  be  used  by  a  student 
to  request  withdrawal  from  further  course  work,  to  notify  IPD  of  a 
change  in  address,  to  notify  IPD  about  changes  in  supervisor  status 
and  to  notify  IPD  of  other  changes  as  described  on  the  form. 

To  increase  the  probability  that  this  form  will  be  used,  it  Is 
suggested  that  it  be  bound  In  each  subcourse  lesson  packet. imme¬ 
diately  in  front  of  the  student  subcourse  critique  form.  Also,  It  Is 
suggested  that  it  be  bound  in  each  subcourse  pretest  scoring  guide 
Immediately  in  front  of  the  supervisor  subcourse  critique  form. 

It  is  suggested  that  IPD  maintain  a  file  of  returned  critique 
forms  by  course  number  and  forward  them  on  a  quarterly  basis  to 
whichever  agency  has  overall  responsibility  for  the  SOJET  program. 

That  agency,  if  it  wishes,  can  use  data  on  the  form  to  determine  per¬ 
sonnel  turbulence  rates  for  students  and  supervisors. 

On  a  monthly  basis  IPD  prepares  a  report  describing  the  number 
of  students  currently  enrolled  in  the  Am \y  correspondence  course 
program,  the  number  of  enrollments  during  the  prior  month,  the  number 
of  withdrawals,  and  so  on.  For  the  SOJET  program  It  Is  recommended 
that  a  revised  version  of  this  report  be  maintained  on  a  monthly 
basis.  The  suggested  format  for  the  report  is  shown  In  Figure  12.  This 
report  would  show  for  the  preceding  12  months  the  number  of  students 
enrolled  by  month,  the  number  of  subcourse  completions  per  month,  the 
number  of  inactive  students,  the  number  of  terminations  per  month,  and 
the  number  of  persons  who  requested  withdrawal  from  the  program  per 
month.  This  information  should  be  of  use  both  to  program  managers  and 
to  course  developers,  since  it  does  provide  a  overview  of  enrollment 
and  completion  trends,  and  an  overall  picture  of  the  cost  of  the 
program. 
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NOTIFICATION  OF  CHANGE  OF  STATUS  OR  MAILING  ADDRESS 

Please  return  this  form  to  Institute  for  Professional 
Development  whenever  one  of  the  conditions  listed  below 
occurs. 

1.  SUPERVISOR  STATUS  (Check  all  that  apply) 

1.  _ Supervisor  will  no  longer  be  able  to  perform  supervisory 

duties. 

a.  _ Student  needs  new  supervisor. 

b.  _ New  supervisor  has  been  located.  Supervisor  Registra¬ 

tion  form  _ Is, _  Is  not  attached. 

2.  _ Supervisor's  mailing  address  has  changed  but  he  still  will 

function  as  supervisor.  New  address  Is  Indicated  below. 

2.  STUDENT  STATUS  (Check  all  that  apply) 

1.  Student  requests  to  withdraw  from  further  course  work, 
fceason  Is  Indicated  below. 

2.  _ Student  Is  changing  unit  but  wishes  to  continue  course 

work.  New  unit  address  Is  Indicated  below. 

3.  _ Student's  mailing  address  has  changed  but  student  remains 

In  same  assignment.  New  address  Indicated  below. 

4.  _ Other  change.  Describe  below 


4.  Student's 

Name _ SSN 


5.  Supervisor's 

Name _ SSN _ 

Test  Form  9  (Revised)  Oct  79 

FIGURE  11.  SUGGESTED  FORM  FOR  06TAINING  INFORMATION  ABOUT  STUDENT 
AND  SUPERVISOR  STATUS  AND  MAILING  ADDRESS  CHANGES 
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VII.  APPLICATION  OF  SOJET  PROCEDURES  TO  CORRESPONDENCE  COURSES 


Throughout  this  research,  features  of  the  SOJET  delivery  system 
were  evaluated  in  terms  of  field  acceptance,  administrative  feasibi¬ 
lity  and  costs.  Modifications  to  the  program  were  made,  as  necessary 
to  debug  the  program(Annex  B).  Additional  modifications  have  been 
proposed  for  future  implementation.  A  point  now  has  been  reached  when 
others  must  decide  whether  the  SOJET  program  is  to  continue  and,  if 
so,  in  what  form.  The  observations  that  follow  are  offered  so  that 
they  may  be  of  assistance  to  those  who  must  decide  about  the  future  of 
the  SOJET  program. 


A.  TYPE  OF  COURSE  TO  WHICH  THE  SOJET  PROCEDURES  APPLY 

The  SOJET  program  was  intended  to  provide  training  support  to 
supervisors  conducting  duty  position  OJT.  Therefore,  SOJET  proce¬ 
dures  are  most  suited  to  courses  that  have  the  following 
characteristics: 

a.  The  instructional  material  is  task-oriented. 

b.  The  instructional  material  is  duty  position-oriented. 

c.  The  lesson  material  covers  tasks  performed  by  specific  job 
encumbents. 

d.  The  subcourses  can  be  studied  in  any  order. 

c.  The  assessment  of  student  performance  can  best  be 
accompl  islied  by  the  use  of  performance  tests. 

f.  Student  performance  cun  be  assessed  using  specially 
prepared  test  scoring  guides. 

There  seem  to  be  no  advantages  to  applying  SOJET  procedures  to 
courses  that  teach  knowledges  as  opposed  to  skills.  Traditional 
correspondence  course  procedures  probably  are  most  cost  effective  for 
presenting  such  courses.  On  the  other  hand,  there  are  many  job  posi¬ 
tions  that  require  mastery  of  a  variety  of  clerical  or  administrative 
skills.  For  such  positions  it  may  be  advantageous  to  develop  SOJET- 
1  ike  courses,  especially  if  special  equipment  or  facilities  are  not 
required  in  support  of  the  courses. 
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B.  COST  OF  SOJET  PROGRAM  DELIVERY  SYSTEM 


It  has  been  estimated  that  the  revised  SOJET  program  delivery 
system  is  approximately  32%  more  costly  than  those  procedures  used  to 
administer  traditional  correspondence  courses  (Table  17).  As  noted 
already,  most  of  the  increased  cost  can  be  attributed  to  the  need  to 
print,  store,  retrieve  and  mail  subcourse  test  scoring  guides.  Any 
delivery  system  that  employs  such  guides  will  incur  similar  costs. 

When  the  cost  of  the  delivery  system  for  the  SOJET  program  and  for 
traditional  correspondence  courses  are  equated  for  printing  and 
mailing  costs  ($43.68  vs.  $40.91}  there  is  only  a  seven  percent  cost 
difference  between  the  two  approaches.  This  difference  is  almost 
entirely  due  to  the  added  labor  required  to  store  and  retrieve  separa¬ 
tely  packaged  lesson  and  test  material  (SOJET  program). 

The  benefits  which  might  be  expected  to  offset  the  added  SOJET 
program  costs  could  not  be  precisely  identified,  but  they  seem  to  include. 

1)  The  "one-shot"  mailing  approach  results  in  the  distribution 
of  lesson  material  that  can  be  used  as  reference  material 
by  persons  not  enrolled  or  registered  in  the  program 
(Table  34). 

2)  Students  are  able  to  study  subcoursc  material  in  an  order 
that  meet  their  training  needs  and  those  of  their  unit 
(Table  32). 

3)  A  reduction  in  the  time  required  to  conduct  on-the-job 
training,  as  reported  by  course  supervisors  (Table  23). 


C.  IMPACT  OF  PERSONNEL  TURBULENCE 

Within  active  An;y  units  personnel  turbulence  is  fairly  high,  and 
this  can  have  a  negative  Impact  on  student  progress  in  most  corres¬ 
pondence  courses.  The  impact  is  apt  to  be  especially  severe  for  duty 
position  courses  which  require  Involvement  of  the  duty  position  super¬ 
visor.  In  addition  to  being  task-oriented  and  duty  position-oriented, 
SOJET  lesson  material  was  packaged  into  16  courses,  the  four  most 
popular  of  which  contained  16  to  19  subcourses  (Table  1).  It  Is 
highly  likely  that  at  least  50%  of  active  Army  students  enrolled  in 
either  of  these  four  course  would  be  reassigned  before  they  had  the 
opportunity  to  complete  their  study  program  (Table  14,  page  34).  Once 
reassigned,  they  would  have  little  incentive  for  continuing  their 
study  unless  their  new  job  still  was  within  an  intelligence  (S2)  or 
operations  (S3)  section. 

Personnel  turbulence  probably  contributed  to  the  high  "no  start" 
rate  for  active  Army  students.  However,  those  students  wtio  submitted 
one  or  more  tests,  especially  active  Army  students,  completed  a  high 
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percent  of  their  enrol  led- for  program  of  study  (Table  11).  This  last 
finding  suggests  the  SOJET-like  courses  may  be  appropriate  for  high 
turbulence  duty  positions.  The  task-oriented,  self-contained  nature 
of  SOJET  subcourses  allows  a  student  to  benefit  from  study  of  portions 
of  a  course  even  though  he  may  never  have  an  opportunity  to  complete 
an  entire  course. 

SOJET  program  procedures  mandate  that  each  student  have  a 
registered  course  supervisor.  What  indirect  evidence  there  is 
suggests  that  the  loss  of  a  course  supervisor  does  not  greatly  affect 
student  progress.  The  SOJET  system  was  designed  so  that  students 
could  obtain  a  new  supervisor  easily.  Forms  were  provided  for 
accomplishing  this,  and  during  this  study  20  formal  requests  were  made 
to  change  course  supervisors.  In  addition  to  these  formal  requests,  a 
student  could  find  a  person  to  act  for  his  original  course  supervisor 
without  reporting  this  to  IPD.  Co>  "rsatlons  with  active  Anqy  stu¬ 
dents  and  supervisors  suggested  th  this  did  occur  on  occasion. 


VIII.  SUMMARY  AND  CONCLUSIONS 


A.  COMPLETION  OF  SOJET  SYSTEM  DESIGN 

During  this  study  the  assigned  objectives  were  accomplished. 
Deficiencies  In  the  original  SOJET  delivery  system  were  Identified  and 
corrective  procedures  were  developed.  Some  of  these  procedures  have 
been  Implemented.  In  addition,  -the  design  of  the  SOJET  program's 
delivery  system  was  completed.  This  Included:  (a)  the  development  of 
central  management  procedures  which  could  be  Implemented  without  using 
special  data  files  and  management  procedures;  and  (b)  the  development 
of  procedures  for  preparing  a  variety  of  feedback  reports  for  use  by 
course  developers.  The  description  of  the  revised  SOJET  program  deli¬ 
very  system  (Annex  B)  can,  with  minimum  programming  assistance,  be 
Implemented  using  management  and  data  processing  procedures  soon  to  be 
adopted  by  IPD. 


B.  EVALUATION  OF  SOJET  DELIVERY  SYSTEM 


Acceptance  of  Role  of  Course  Supervisor 

In  the  process  of  refining  and  completing  the  design  of  the  SOJET 
program  delivery  system,  the  management  system  objectives  Incorporated 
Into  the  program  and  the  approaches  taken  to  achieve  these  objectives 
were  evaluated. 

It  was  found  that  both  students  and  supervisors  were  willing  to 
have  course  supervisors  participate  1r,  extension  training.  More 
specifically,  course  supervisors  were  willing  to  register  by  name; 
help  students  enroll  and  plan  their  program  of  study;  assume  respon¬ 
sibility  for  test  security;  and  assume  responsibility  for  admi¬ 
nistrating  and  scoring  tests,  reviewing  test  results  with  students, 
and  reporting  the  results  to  IPD.  However,  supervisors  expressed  con¬ 
cern  about  the  time  required  to  accomplish  these  actions,  and  tended 
to  complain  about  the  complexity  of  the  original  SOJET  enrollment  and 
test  score  reporting  procedures.  On  the  basis  of  these  findings, 
revised  and  simplified  procedures  for  enrollment  and  for  test 
reporting  were  developed  and  implemented. 

The  requirement  to  have  a  registered  course  supervisor  Increases 
program  administrative  costs.  On  the  other  hand,  course  supervisors 
help  students  Identify  training  requirements,  provide  a  source  of  aid 
to  students  and  provide  feedback  to  students  on  test  results.  Also, 
they  may,  on  occasion,  motivate  students  to  continue  In  their  study 
program.  The  beneficial  ejects  of  these  and  rther  supervisor  actlvl- 


ties  could  not  be  Identified  during  this  research.  Future  versions  of 
the  SOJET  program  can  continue  to  require  a  course  supervisor.  As  an 
alternative,  test  material  can  be  sent  to  a  student  who  then  passes  It 
on  to  a  person  who  agrees  to  safeguard  the  material  and  to  administer 
and  score  tests  (the  Supervised  On-the-Job  Training— SOJT— approach). 
Which  option  eventually  is  selected  is  a  matter  for  the  Amy  to  decide. 


Impact  on  OJT  Time  Requirements 

It  was  anticipated  that  the  SOJET  program  would  provide  training 
support  to  the  supervisor  and  that  this  support  would  lessen  the  time 
required  of  the  supervisor  to  conduct  on-the-job  training.  Survey 
data  from  course  supervisors  Indicated  that  this  was  the  case.  Most 
course  supervisors  did  report  that  the  lesson  material  was  of  good 
quality  and  was  job  relevant,  and  that  requirements  for  OJT  were 
reduced  as  the  result  of  participating  In  the  SOJET  program.  However, 
certain  features  of  the  program  did  not  lead  to  an  expected  reduction 
in  the  demands  on  supervisor  time.  In  particular,  the  Student 
Training  Plan,  which  was  Intended  to  help  supervisors  and  students 
identify  training  requirements,  was  completed  In  a  cursory  manner  by 
many  supervisors.  Because  of  this,  and  In  the  Interest  of  reducing 
the  time  needed  to  enroll  in  the  program,  the  requirement  to  complete 
and  return  a  Student  Training  Plan  as  part  of  the  enrollment  applica¬ 
tion  was  discontinued.  STPs  still  are  provided  In  the  enrollment 
packet  because  a  minority  of  applicants  apparently  find  them  quite 
useful.  It  Is  suggested,  however,  that  STPs  no  longer  be  distributed 
once  the  list  of  SOJET  subcourses  has  been  published  In  all  combat 
arms  school  catalogs. 


Impact  of  M0ne-Shot"  Mailing  Approach 

The  None*shotN  mailing  approach  used  with  the  SOJET  program  pro¬ 
vides  students  and  supervisors  with  a  means  for  sequencing  training  to 
fit  local  requirements.  Survey  data  and  test  submission  sequence  data 
both  showed  that  about  one-third  of  the  students  planned  their 
sequence  of  subcourse  study  on  the  basis  of  their  own  training  need  or 
unit  requirements.  The  remaining  students  studied  subcourses  In  the 
order  in  which  thqy  were  numbered,  or  In  accordance  with  other  con¬ 
siderations  irrelevant  to  unit  needs.  The  “one-shotM  mailing  approach 
also  provided  units  with  a  complete  set  of  task-oriented  lesson 
material.  Many  students  and  supervisors  reported  that  their  lesson 
material  was  used  by  others  as  reference  aids  or  for  preparing  for 
Skill  Qualification  Tests.  Because  of  mailing  costs  the  "one-shot1* 
mailing  approach  Is  somewhat  more  expensive  than  that  followed  with 
other  correspondence  courses.  However,  its  continued  use  Is 
suggested. 


Adherence  to  Testing  Procedures 


Test  Scoring  Guides.  One  objective  of  the  SOJET  program  was  to 
provide  students  immediate  feedback  about  test  results.  To  make  this 
possible,  test  scoring  guides  usable  by  non-subject  matter  experts 
were  developed.  Based  on  survey  responses  it  appeared  that  most  su¬ 
pervisors  had  little  difficulty  using  these  guides.  However,  a  number 
of  supervisors  noted  that  the  scoring  guides  contained  printing  and 
collation  errors.  This  sometimes  made  it  difficult  to  match  the  guide 
material  with  test  questions.  All  these  problems  should  be  corrected 
before  the  next  printing  of  SOJET  course  test  and  scoring  guide 
material. 

Student  Feedback.  The  course  supervisors  were  requested  to 
score  the  tests  and  review  test  results  with  students  as  soon  as 
possible  following  test  administration.  Host  students  and  supervisors 
reported  that  this  requirement  was  followed,  especially  within  active 
Army  units.  This  is  more  difficult  to  accomplish  without  time  lags  In 
National  Guard  or  Reserve  units  because  such  units  meet  only  periodi¬ 
cally. 

Test  Reporting  Requirements.  Most  features  of  the  SOJET  program 
dealing  with  the  collection  and  reporting  of  feedback  data  for  course 
developers  were  designed  during  the  conduct  of  this  study.  Therefore, 
the  usefulness  of  these  data  were  not  evaluated.  One  feature  that  was 
assessed  was  the  requirement  to  report  failing  as  well  as  passing  pre- 
and  posttest  scores  to  IPO.  Analysis  of  training  records  suggested 
that  supervisors  were  reluctant  to  report  failing  scores,  especially 
pretest  failures.  In  view  of  this  finding,  and  the  reported  need  to 
simplify  the  test  reporting  process,  the  requirement  to  report  pretest 
scores  as  soon  as  the  pretests  were  taken  was  discontinued.  Now, 
revised  SOJET  procedures  for  recording  and  reporting  test  scores 
require  that  both  pre-  and  posttost  results  be  recorded  after  the  stu¬ 
dent  has  successfully  passed  all  subcoursc  test  requirements.  Also, 
test  results  now  are  reported  on  the  standard  test  reporting  form  used 
with  other  correspondence  courses  instead  of  on  a  special  form 
designed  for  the  SOJI'T  program.  These  revised  requirements  and  proce¬ 
dures  arc  much  less  time  consuming  for  both  course  supervisors  and  IPO 
personnel . 


C.  STOOCHT  PROGRESS  AND  PERSONNEL  TURBULENCE 

Analysis  of  student  training  records  revealed  that  70S  and  51S  of 
AA  and  NG/USAR  students,  respectively,  had  never  submitted  a  single 
subcourse  test  (Tabic  8).  Overall,  the  course  completion  rate  for  all 
students  who  have  been  enrolled  tor  at  least  four  ninths  (classes  1 
through  6)  was  approximately  11  percent  (calculated  from  Table  8 
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data).  On  the  other  hand,  analysis  of  student  progress  in  terms  of 
the  number  of  subcourses  completed  (Table  10)  revealed  that  of  those 
students  who  submitted  one  or  more  tests,  active  Arcy  students  had 
completed  12  subcourses  (COX  of  their  training  program)  by  the  end  of 
six  months  following  enrollment.  The  rate  of  progress  tended  to  be 
considerably  less  for  NG/USAR  students. 

Data  on  personnel  turbulence  (Table  14)  suggested  that  active 
Any  personnel  can  anticipate  reassignment  after  six  months.  For  many 
students  their  new  duties  will  not  be  related  to  operations  or 
intelligence,  and  therefore  they  probably  will  discontinue  their  SOJET 
program. 

The  foregoing  data  suggest  that  personnel  turbulence  had  a  nega¬ 
tive  iupact  on  student  progress,  especially  If  student  progress  Is 
defined  in  terms  of  course  completion  percentages.  When  student 
progress  is  defined  in  terms  of  subcourse  completion  percentages  it 
appears  that  personnel  turbulence  had  less  of  a  negative  impact. 
Probably  this  was  because  the  task-oriented,  self-contained  nature  of 
each  SOJET  subcourse  made  it  possible  to  benefit  from  the  study  of  one 
or  a  few  subcourses  even  though  an  entire  course  was  not  completed. 


0.  COST  OF  SOJET  PROGRAM 

A  cost  comparison  of  SOJCT  and  traditional  procedures  for  the 
delivery  and  central  management  of  correspondence  courses  and/or 
extension  training  indicated  that  SOJET  procedures  (revised)  were  321 
more  costly  (Table  17).  It  was  determined  that  a  substantial  portion 
of  the  increased  cost  of  the  SOJET  program  could  be  attributed  to  the 
need  for  performance  touts  and  test  scoring  guides,  material  required 
b>  any  program  that  depends  on  field  personnel  to  administer  and  score 
performance  tests.  When  these  costs  arc  equalized  for  SOJET  ami  for 
traditional  correspondence  programs  it  was  found  that  SOJET  procedures 
were  only  7%  wore  costly  than  those  employed  to  delivery  traditional 
correspondence  programs  (page  88).  It  was  concluded  that  the  extra 
costs  associated  with  the  SOJET  procedures  need  not  be  a  deterrent  to 
a;  plying  them  to  selected  courses,  especially  those  that  are  designed 
to  be  task-  and  duty  pos it  ion-oriented. 


E.  CONTINUANCE  CF  SOJO  PROGRAM 

Any  decision  to  discontinue  the  SOJET  progta-t  utst  take  into 
account  two  important  factors.  First,  there  is  a  need  for  instruc¬ 
tional  material  for  battalion  S2  and  S3  NCQ  duty  positions.  These 
positions  were  selected  as  the  vehicle  for  the  SOJET  program  in  order 
to  fill  an  instructional  void.  The  need  for  instructional  material 
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for  these  duty  positions  was  substantiated  during  Interviews  with 
active  Amy  personnel.  Secondly,  the  extra  cost  of  using  SOJET  proce¬ 
dures  (revised)  to  manage  the  16  SOJET  program  courses  was  calculated 
as  $6,288  ($10.48x600  )  for  an  annual  enrollment  of  600  students  (see 
Table  17).  Based  on  these  two  findings,  it  was  concluded  that  the 
SOJET  courses  should  continue  to  be  offered  by  the  ACCP  and  should 
continue  to  be  administered  using  SOJET  procedures  for  at  least 
through  calendar  year  1980.  This  would  provide  time  to  integrate 
SOJET  procedures  (final)  with  those  under  adoption  as  part  of  IPO's 
IMIS  system.  Also,  it  would  provide  time  to  collect  additional 
enrollment  and  training  data  (test  submission  data)  which  could  be 
used  to  evaluate  the  long-term  acceptance  and  effectiveness  of  the 
program.  If  this  last  suggestion  is  adopted,  it  1$  proposed  that,  for 
comparative  purposes,  similar  types  of  enrollment  and  student  progress 
data  be  collected  for  two  or  three  correspondence  courses  which  use 
more  traditional  delivery  procedures.  Also,  data  should  be  collected 
on  courses  administered  using  IPO's  Supervised  On-the-Job  Training 
procedures. 


APPENDIX  A 


TABLE  A-l 


DISTRIBUTION  OF  ENROLLMENTS  IN  TERMS  OF  CURRENT  DUTY  POSITION3 

(Senior  Sergeant  Courses) 


Course 

Component 

OPS  SGT/ 
ASST  OPS 
SGT 

Current 

INTEL  SGT/ 
ASST  INTEL 
SGT 

Duty  Position 

OPS  INTEL 
ASST/  ASST/ 
SPEC  SPEC 

Non  OPS/ 

INTEL 

POSITIONS 

Total 

OPS  SGT 

AR  (Y01)C 

AA 

21 

1 

6 

28 

NG/USAR 

49 

4 

2 

- 

23 

78 

INF  (Y02) 

AA 

18 

1 

5 

24 

NG/USAR 

59 

3 

1 

13 

76 

FA. (Y03) 

AA 

11 

_ 

22 

33 

NG/USAR 

36 

4 

- 

10 

50 

ADA  (Y04) 

AA 

4 

_ 

• 

3 

7 

NG/USAR 

4 

- 

1 

- 

5 

INTEL  SGT 

AR  (Y05) 

AA 

i 

9 

• 

X 

2 

13 

NG/USAR 

i 

25 

5 

- 

2 

33 

INF  (Y06) 

AA 

. 

5 

• 

• 

4 

9 

NG/USAR 

2 

35 

- 

1 

3 

41 

FA  (Y04) 

AA 

w. 

8 

• 

3 

11 

NG/USAR 

1 

21 

* 

i 

- 

1 

24 

ADA  (Y08) 

AA 

• 

2 

• 

• 

2 

NG/USAR 

- 

1 

- 

- 

- 

1 

Selected 

PI 

Sub- 

AA 

7 

m 

m 

2 

21 

Courses** 

NG/USAR 

Bfl 

11 

- 

2 

4 

36 

(Y01-Y08) 

■H... 

Total 

AA 

1  m 

33 

i 

47 

148 

NG/USAR 

mm 

3 

56 

344 

Enrollment  as  of  1  September  1979 

Persons  who  enrolled  In  less  than  a  full  course 


cAr  *  Armor;  INF  *  Infantry;  FA  *  Field  Artillery;  ADA  «  Air  Defense  Artillery 
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.  TABLE  A-l  (Cont'd) 

DISTRIBUTION  OF  ENROLLMENTS  IN  TERMS  OF  CURRENT  DUTY  POSITION® 
(Junior  Sergeant/Speclallst  Courses) 


Course 

Component 

OPS  SGT/ 
ASST  OPS 
SGT 

Current  Duty  Position 

INTEL  SGT/  OPS  INTEL 

ASST  INTEL  ASST/  ASST/ 

SGT  SPEC  SPEC 

* 

Non  OPS/ 

INTEL 

POSITIONS 

Total 

OPS  S6T 

AA 

• 

_ 

7 

3 

3 

13 

NG/USAR 

1 

- 

13 

8 

7 

29 

INF  (Y10) 

AA 

• 

8 

8 

NG/USAR 

1 

- 

14 

- 

- 

15 

FA  (Yll) 

AA 

1 

• 

3 

4 

NG/USAR 

- 

- 

3 

- 

2 

5 

ADA  (Y12) 

AA 

1 

5 

• 

6 

NG/USAR 

- 

- 

5 

m 

m 

5 

INTEL  S6T 

AR  (Y13) 

AA 

_ 

• 

i 

7 

5 

13 

NG/USAR 

2 

3 

14 

- 

19 

INF  (Y14) 

AA 

• 

• 

NG/USAR 

- 

- 

3 

1 

4 

FA  (Y15) 

AA 

• 

• 

_ 

m 

• 

• 

NG/USAR 

- 

1 

- 

i 

1 

3 

AA 

• 

. 

• 

• 

• 

NG/USAR 

- 

1 

- 

1 

i 

3 

Selected 

Sub- 

AA 

• 

1 

2 

1 

2 

6 

Courses” 

(Y01-Y08) 

NG/USAR 

1 

- 

1 

7 

- 

9 

Total 

AA 

1 

2 

26 

u 

10 

50 

NG/USAR 

3 

4 

39 

34 

12 

92 

®Enrollment  as  of  1  September  1979 

Persons  who  enrolled  in  less  than  a  full  course 


cAr  *  Armor;  INF  ■  Infantry;  FA  ■  Field  Artillery;  ADA  *  Air  Defense  Artillery 
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TABLE  A- 2 


DISTRIBUTION  OF  ENROLLMENTS  IN  TERMS  OF  CURRENT  SKILL  LEVEL3 

(Senior  Sergeant  Courses) 


Course 

Component 

1 

2 

Current  Skill  Level 

3  4  5 

Unknown 

Total 

OPS  SGT 

AR  (Y01) 

AA 

2 

2 

5 

15 

3 

1 

28 

NG/USAR 

3 

2 

15 

32 

21 

5 

78 

INF  (Y02) 

AA 

. 

7 

4 

12 

1 

m 

24 

NG/USAR 

- 

5 

12 

21 

36 

2 

76 

FA  (Y03) 

AA 

. 

1 

9 

19 

3 

1 

33 

NG/USAR 

- 

- 

6 

15 

29 

- 

50 

ADA  (Y04) 

AA 

1 

1 

1 

4 

• 

7 

NG/USAR 

- 

- 

1 

2 

2 

- 

5 

INTEL  SGT 

AR  (Y05) 

AA 

i 

. 

7 

2 

• 

* 

* 

13 

NG/USAR 

- 

- 

9 

12 

10 

2 

33 

INF  (Y06) 

AA 

1 

5 

1 

2 

. 

9 

NG/USAR 

3 

11 

13 

14 

- 

41 

FA  (Y07) 

AA 

2 

3 

6 

• 

11 

NG/USAR 

- 

3 

7 

14 

24 

AA 

1 

• 

m 

1 

2 

NG/USAR 

- 

- 

m 

1 

- 

1 

Selected 

Sub- 

AA 

i 

5 

4 

3 

5 

3 

21 

Courses" 

(Y01-Y08) 

NG/USAR 

i 

1 

2 

11 

20 

1 

36 

Total 

AA  1 

5 

20 

38 

62 

14 

9 

148 

4 

11 

59 

113 

147 

10 

344 

^Persons  who  enrolled  In  less  than  a  full  course 
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TABLE  A-2  (Cont'd) 


DISTRIBUTION  OF  ENROLLMENTS  IN  TERMS  OF  CURRENT  SKILL  LEVEL 
(Junior  Sergeant/Specialist  Courses) 


Course 

Component 

1 

.  2 

Current  Skill 

3  4 

Level 

5 

Unknown 

Total 

OPS  SGT 

AA 

9 

3 

1 

m 

m 

13 

NG/USAR 

12 

7 

4 

• 

1 

5 

29 

AA 

7 

1 

_ 

_ 

8 

NG/USAR 

8 

1 

5 

1 

15 

FA  (Yll) 

AA 

3 

1 

. 

4 

NG/USAR 

3 

1 

1 

- 

5 

AA 

6 

_ 

6 

NG/USAR 

- 

3 

1 

1 

5 

INTEL  SGT 

AR  (Y13) 

AA 

10 

1 

2 

• 

13 

NG/USAR 

3 

8 

7 

1 

19 

INF  (Y14) 

AA 

• 

• 

_ 

• 

NG/USAR 

1 

2 

i 

4 

FA  (Y15) 

AA 

• 

. 

• 

. 

NG/USAR 

1 

2 

- 

3 

ADA  (Y16) 

AA 

m 

• 

_ 

NG/USAR 

1 

2 

- 

3 

Selected 

Sub- 

AA 

5 

1 

. 

- 

• 

6 

Courses'1 

(Y09-Y16) 

NG/USAR 

2 

4 

1 

2 

- 

9 

Total 

AA 

Mi 

n 

3 

m 

m 

50 

B9 

El 

20 

5 

1 

5 
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TABLE  A-3 


DISTRIBUTION  OF  ENROLLMENTS  IN  TERMS  OF  CURRENT  MOS* 
(Senior  Sergeant  Courses) 


Combat  MOS 

Non-Combat  MOS 

Course 

Component 

AR 

INF 

FA 

ADA 

ENG 

INTEL 

ORD 

OTHER 

UNK 

Total 

OPS  SGT 

• 

■ 

AR  (Y01) 

AA 

8 

10 

6 

2 

1 

- 

- 

- 

1 

28 

NG/USAR 

45 

6 

6 

- 

4 

4 

4 

2 

5 

78 

INF  (Y02) 

AA 

1 

21 

• 

• 

1 

• 

.. 

1 

• 

24 

NG/USAR 

1 

56 

- 

- 

3 

3 

5 

8 

- 

76 

FA. (Y03) 

AA 

• 

30 

1 

1 

• 

• 

1 

33 

NG/USAR 

- 

9 

36 

- 

- 

- 

5 

fl 

50 

ADA  (Y04) 

AA 

.. 

• 

• 

7 

•• 

• 

.. 

m 

■ 

7 

NG/USAR 

- 

- 

- 

5 

- 

- 

- 

- 

B 

5 

INTEL  SGT 

AR  (Y05) 

AA 

3 

3 

2 

2 

2 

1 

- 

- 

m 

13 

NG/USAR 

18 

3 

2 

- 

- 

8 

- 

1 

m 

33 

INF  (Y06) 

AA 

• 

5 

1 

1 

2 

.. 

m 

9 

NG/USAR 

20 

19 

- 

- 

- 

1 

1 

m 

41 

FA  (Y07) 

AA 

• 

7 

1 

1 

2 

.. 

m 

11 

NG  USAR 

- 

- 

20 

- 

- 

3 

1 

m 

24 

ADA  (Y08) 

AA 

. 

• 

. 

1 

. 

1 

- 

m 

2 

NG/USAR 

- 

- 

- 

1 

m 

- 

- 

m 

_ 1_ 

Selected 

AA 

■ 

m 

H 

6 

. 

m 

i 

|| 

21 

Sub- 

NG/USAR 

1 

H 

2 

i 

1 

2 

6 

36 

Courses8 

(Y01-Y08) 

■ 

H 

1 

Total 

AA 

12 

58 

42 

21 

9 

2 

2 

148 

(Y01-Y08) 

85 

97 

83 

8 

8 

20 

15 

22 

344 

^Persons  who  enrolled  in  less  than  a  full  course 


^Eng  ■  Engineering;  Intel  *  Intelligence;  Ord  *  Ordnance 
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TABLE  A-3  (Cont'd) 


DISTRIBUTION  OF  ENROLLMENTS  IN  TERMS  OF  CURRENT  MOS 
(Junior  Sergeant/Specialist  Courses) 


Course 

Component 

AR 

Combat  MOS 

INF  FA 

ADA 

ENG 

Non-Combat  MOS 
INTEL  ORD 

OTHER  UNK 

Total 

OPS  SGT 

■ 

AR  (Y09) 

AA 

- 

2 

- 

5 

3 

2 

1 

- 

13 

NG/USAR 

n 

6 

3 

i 

i 

2 

- 

1 

4 

29 

INF  (Y10) 

AA 

8 

. 

• 

• 

• 

. 

m 

8 

NG/USAR 

- 

14 

- 

- 

- 

“ 

1 

- 

- 

IS 

FA  (Yll) 

AA 

_ 

• 

2 

i 

• 

• 

• 

1 

4 

NG/USAR 

- 

- 

3 

- 

- 

- 

1 

1 

m 

5 

AA 

• 

• 

4 

• 

1 

1 

m 

6 

NG/USAR 

3 

m 

- 

2 

- 

- 

- 

• 

- 

5 

INTEL  ASST 
AR  (Y13) 

AA 

m 

2 

4 

6 

m 

i 

13 

NG/USAR 

2 

2 

3 

- 

- 

ii 

- 

1 

- 

19 

INF  (Y14) 

AA 

. 

• 

m 

• 

. 

. 

• 

_ 

1 

NG/USAR 

- 

1 

- 

- 

- 

3 

- 

- 

4 

FA  (Y15) 

AA 

. 

• 

_ 

_ 

. 

. 

• 

1 

NG  USAR 

- 

1 

- 

m 

1 

- 

1 

3 

ADA  (Y16) 

AA 

• 

• 

• 

• 

• 

m 

. 

]  w* 

NG/USAR 

- 

1 

- 

1 

• 

i 

- 

m 

3 

Selected 

AA 

m 

1 

i 

2 

m 

i 

i 

6 

Sub- 

NG/USAR 

5 

3 

- 

. 

- 

- 

- 

i 

• 

9 

Courses 

AA  1 

H 

11 

5 

16 

r 

ii 

2 

4 

i 

n 

■MM 

UL 

27 

4 

18 

2 

5 

- 

UL. 

I 

o 

§ 


CO 

s 


>- 

I— 

E 


CO 


8 

i 


to 

H 

2S 

UJ 


o 

s 

UJ 

a 


03 

0£ 

to 

S 
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dNumbers  represent  percent 


